Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 13:59:13.243530 16530 test_util.cc:268] Using random seed: -466156200
I20240501 13:59:13.255908 16530 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:13.255997 16530 ts_itest-base.cc:113] --------------
I20240501 13:59:13.256042 16530 ts_itest-base.cc:114] 8 tablet servers
I20240501 13:59:13.256103 16530 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:13.256170 16530 ts_itest-base.cc:116] --------------
2024-05-01T13:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:13Z Disabled control of system clock
I20240501 13:59:13.281293 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:43309
--webserver_interface=127.16.36.190
--webserver_port=0
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.190:43309
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 13:59:13.621340 16542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:13.621690 16542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:13.621778 16542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:13.629508 16542 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:13.629647 16542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:13.629694 16542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:13.629752 16542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:13.629802 16542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:13.639300 16542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.190:43309
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:43309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:13.640671 16542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:13.642515 16542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:13.654480 16547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.655025 16548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.655537 16550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:13.656138 16542 server_base.cc:1034] running on GCE node
I20240501 13:59:13.656985 16542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:13.658612 16542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:13.659862 16542 hybrid_clock.cc:648] HybridClock initialized: now 1714571953659810 us; error 52 us; skew 500 ppm
I20240501 13:59:13.660218 16542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:13.665993 16542 webserver.cc:460] Webserver started at http://127.16.36.190:33845/ using document root <none> and password file <none>
I20240501 13:59:13.666885 16542 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:13.667016 16542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:13.667357 16542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:13.670159 16542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "42091c0ca883447badbb280302221537"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-9q9t"
I20240501 13:59:13.671028 16542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "42091c0ca883447badbb280302221537"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-9q9t"
I20240501 13:59:13.677070 16542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240501 13:59:13.680838 16556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.682296 16542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:13.682499 16542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
uuid: "42091c0ca883447badbb280302221537"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-9q9t"
I20240501 13:59:13.682770 16542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:13.713976 16542 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:13.715083 16542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:13.715488 16542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:13.745100 16542 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.190:43309
I20240501 13:59:13.745177 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.190:43309 every 8 connection(s)
I20240501 13:59:13.746704 16542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:13.750305 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 16542
I20240501 13:59:13.750584 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:13.750763 16608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:13.761451 16608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537: Bootstrap starting.
I20240501 13:59:13.764647 16608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:13.766137 16608 log.cc:824] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:13.769467 16608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537: No bootstrap required, opened a new log
I20240501 13:59:13.776217 16608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42091c0ca883447badbb280302221537" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43309 } }
I20240501 13:59:13.776618 16608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:13.776764 16608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42091c0ca883447badbb280302221537, State: Initialized, Role: FOLLOWER
I20240501 13:59:13.777562 16608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42091c0ca883447badbb280302221537" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43309 } }
I20240501 13:59:13.777827 16608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:13.777956 16608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:13.778113 16608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:13.781322 16608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42091c0ca883447badbb280302221537" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43309 } }
I20240501 13:59:13.781903 16608 leader_election.cc:302] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42091c0ca883447badbb280302221537; no voters:
I20240501 13:59:13.782480 16608 leader_election.cc:288] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:13.782681 16613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:13.783296 16613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [term 1 LEADER]: Becoming Leader. State: Replica: 42091c0ca883447badbb280302221537, State: Running, Role: LEADER
I20240501 13:59:13.783982 16613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42091c0ca883447badbb280302221537" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43309 } }
I20240501 13:59:13.784907 16608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:13.787461 16614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42091c0ca883447badbb280302221537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42091c0ca883447badbb280302221537" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43309 } } }
I20240501 13:59:13.787902 16614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:13.788411 16615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42091c0ca883447badbb280302221537. Latest consensus state: current_term: 1 leader_uuid: "42091c0ca883447badbb280302221537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42091c0ca883447badbb280302221537" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43309 } } }
I20240501 13:59:13.788846 16615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:13.789566 16618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:13.795997 16618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:13.805987 16618 catalog_manager.cc:1347] Generated new cluster ID: 178383fababb4d8388a95df2aa36832c
I20240501 13:59:13.806131 16618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:13.828879 16618 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:13.830157 16618 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:13.841624 16618 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 42091c0ca883447badbb280302221537: Generated new TSK 0
I20240501 13:59:13.842823 16618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:13.857221 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.129:0
--local_ip_for_outbound_sockets=127.16.36.129
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 13:59:14.216745 16632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:14.217090 16632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:14.217229 16632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:14.225121 16632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:14.225261 16632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:14.225431 16632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.129
I20240501 13:59:14.235559 16632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.129
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:14.237003 16632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:14.238941 16632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:14.252445 16637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.253993 16632 server_base.cc:1034] running on GCE node
W20240501 13:59:14.254456 16640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.254488 16638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.255344 16632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:14.256444 16632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:14.257665 16632 hybrid_clock.cc:648] HybridClock initialized: now 1714571954257602 us; error 54 us; skew 500 ppm
I20240501 13:59:14.258021 16632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:14.260905 16632 webserver.cc:460] Webserver started at http://127.16.36.129:44253/ using document root <none> and password file <none>
I20240501 13:59:14.261770 16632 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:14.261896 16632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:14.262225 16632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:14.265157 16632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "10f8b1516d5142709756d06b06c5d55f"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-9q9t"
I20240501 13:59:14.266055 16632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "10f8b1516d5142709756d06b06c5d55f"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-9q9t"
I20240501 13:59:14.271747 16632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 13:59:14.275720 16646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.277254 16632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:14.277448 16632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "10f8b1516d5142709756d06b06c5d55f"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-9q9t"
I20240501 13:59:14.277698 16632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:14.313755 16632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:14.314777 16632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:14.315063 16632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:14.316102 16632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:14.317747 16632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:14.317850 16632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.317973 16632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:14.318046 16632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.351152 16632 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.129:35177
I20240501 13:59:14.351253 16758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.129:35177 every 8 connection(s)
I20240501 13:59:14.352628 16632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:14.356374 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 16632
I20240501 13:59:14.356626 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:14.360188 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.130:0
--local_ip_for_outbound_sockets=127.16.36.130
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:14.366549 16759 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:14.366926 16759 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:14.367821 16759 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:14.370327 16573 ts_manager.cc:194] Registered new tserver with Master: 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:14.372908 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.129:54317
W20240501 13:59:14.702282 16763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:14.702656 16763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:14.702807 16763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:14.710681 16763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:14.710829 16763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:14.711010 16763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.130
I20240501 13:59:14.720465 16763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.130
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:14.721894 16763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:14.723831 16763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:14.736387 16768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.737097 16771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.737859 16769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.738417 16763 server_base.cc:1034] running on GCE node
I20240501 13:59:14.739149 16763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:14.740190 16763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:14.741420 16763 hybrid_clock.cc:648] HybridClock initialized: now 1714571954741366 us; error 54 us; skew 500 ppm
I20240501 13:59:14.741762 16763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:14.744288 16763 webserver.cc:460] Webserver started at http://127.16.36.130:43719/ using document root <none> and password file <none>
I20240501 13:59:14.745328 16763 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:14.745471 16763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:14.745893 16763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:14.749577 16763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d871186c2b234914ab63dc5e136f0f03"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-9q9t"
I20240501 13:59:14.750655 16763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d871186c2b234914ab63dc5e136f0f03"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-9q9t"
I20240501 13:59:14.757850 16763 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20240501 13:59:14.762673 16777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.764596 16763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:14.764786 16763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d871186c2b234914ab63dc5e136f0f03"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-9q9t"
I20240501 13:59:14.765035 16763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:14.779598 16763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:14.780474 16763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:14.780732 16763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:14.781682 16763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:14.783528 16763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:14.783620 16763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.783707 16763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:14.783761 16763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.815626 16763 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.130:42745
I20240501 13:59:14.815685 16889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.130:42745 every 8 connection(s)
I20240501 13:59:14.817077 16763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:14.819242 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 16763
I20240501 13:59:14.819432 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:14.823045 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.131:0
--local_ip_for_outbound_sockets=127.16.36.131
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:14.831210 16890 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:14.831580 16890 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:14.832530 16890 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:14.834256 16573 ts_manager.cc:194] Registered new tserver with Master: d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:14.835117 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.130:43635
W20240501 13:59:15.159266 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.159605 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.159761 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.167253 16893 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:15.167438 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.167618 16893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.131
I20240501 13:59:15.176844 16893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.131
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.178225 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.180002 16893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.193341 16898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.193909 16899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.194387 16901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.195168 16893 server_base.cc:1034] running on GCE node
I20240501 13:59:15.195849 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.196877 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.198084 16893 hybrid_clock.cc:648] HybridClock initialized: now 1714571955198021 us; error 64 us; skew 500 ppm
I20240501 13:59:15.198407 16893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.201066 16893 webserver.cc:460] Webserver started at http://127.16.36.131:37773/ using document root <none> and password file <none>
I20240501 13:59:15.201876 16893 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.202003 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.202332 16893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.205189 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f4b54fe4701746c0930c9b0667c90519"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-9q9t"
I20240501 13:59:15.206061 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f4b54fe4701746c0930c9b0667c90519"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-9q9t"
I20240501 13:59:15.211938 16893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20240501 13:59:15.215807 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.217264 16893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:15.217445 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f4b54fe4701746c0930c9b0667c90519"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-9q9t"
I20240501 13:59:15.217670 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.235118 16893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.236048 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.236338 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.237381 16893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.239076 16893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.239183 16893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.239300 16893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.239372 16893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.272704 16893 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.131:37163
I20240501 13:59:15.272778 17019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.131:37163 every 8 connection(s)
I20240501 13:59:15.274076 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:15.282022 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 16893
I20240501 13:59:15.282254 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:15.285923 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:0
--local_ip_for_outbound_sockets=127.16.36.132
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:15.288578 17020 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:15.288981 17020 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.290141 17020 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:15.291882 16573 ts_manager.cc:194] Registered new tserver with Master: f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:15.292706 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.131:48613
I20240501 13:59:15.376415 16759 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
W20240501 13:59:15.637502 17023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.637830 17023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.637972 17023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.645670 17023 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:15.645817 17023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.645996 17023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 13:59:15.655762 17023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.657138 17023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.659024 17023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.671386 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.671977 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.672413 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.672824 17023 server_base.cc:1034] running on GCE node
I20240501 13:59:15.673485 17023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.674499 17023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.675729 17023 hybrid_clock.cc:648] HybridClock initialized: now 1714571955675679 us; error 70 us; skew 500 ppm
I20240501 13:59:15.676059 17023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.678364 17023 webserver.cc:460] Webserver started at http://127.16.36.132:44559/ using document root <none> and password file <none>
I20240501 13:59:15.679191 17023 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.679313 17023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.679648 17023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.682368 17023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-9q9t"
I20240501 13:59:15.683296 17023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-9q9t"
I20240501 13:59:15.689369 17023 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240501 13:59:15.693351 17037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.695039 17023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 13:59:15.695226 17023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-9q9t"
I20240501 13:59:15.695446 17023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.713357 17023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.714362 17023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.714721 17023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.715832 17023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.717409 17023 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.717512 17023 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.717638 17023 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.717712 17023 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.751693 17023 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:37337
I20240501 13:59:15.751727 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:37337 every 8 connection(s)
I20240501 13:59:15.753160 17023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:15.756268 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 17023
I20240501 13:59:15.756455 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:15.759564 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:0
--local_ip_for_outbound_sockets=127.16.36.133
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:15.766894 17150 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:15.767271 17150 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.768427 17150 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:15.770252 16573 ts_manager.cc:194] Registered new tserver with Master: 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:15.771193 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:38659
I20240501 13:59:15.837867 16890 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
W20240501 13:59:16.112138 17153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:16.112478 17153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:16.112583 17153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:16.120342 17153 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:16.120482 17153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:16.120606 17153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 13:59:16.130081 17153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:16.131487 17153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:16.133273 17153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:16.146534 17158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.146684 17159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.147269 17161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:16.148026 17153 server_base.cc:1034] running on GCE node
I20240501 13:59:16.148681 17153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:16.149686 17153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:16.150940 17153 hybrid_clock.cc:648] HybridClock initialized: now 1714571956150882 us; error 43 us; skew 500 ppm
I20240501 13:59:16.151255 17153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:16.153556 17153 webserver.cc:460] Webserver started at http://127.16.36.133:33141/ using document root <none> and password file <none>
I20240501 13:59:16.154328 17153 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:16.154433 17153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:16.154807 17153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:16.157539 17153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "943a7d1d43144d56b35d62ebf72d1e25"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:16.158356 17153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "943a7d1d43144d56b35d62ebf72d1e25"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:16.163880 17153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:16.167562 17167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.168941 17153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:16.169111 17153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "943a7d1d43144d56b35d62ebf72d1e25"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:16.169324 17153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:16.196520 17153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:16.197500 17153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:16.197857 17153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:16.199007 17153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:16.200822 17153 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:16.200939 17153 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.201066 17153 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:16.201143 17153 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.235715 17153 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:36049
I20240501 13:59:16.235785 17279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:36049 every 8 connection(s)
I20240501 13:59:16.237155 17153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:16.238889 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 17153
I20240501 13:59:16.239070 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:16.242998 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.134:0
--local_ip_for_outbound_sockets=127.16.36.134
--webserver_interface=127.16.36.134
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:16.253666 17280 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:16.253995 17280 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:16.254979 17280 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:16.256558 16573 ts_manager.cc:194] Registered new tserver with Master: 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133:36049)
I20240501 13:59:16.257371 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:42935
I20240501 13:59:16.296027 17020 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
W20240501 13:59:16.598400 17283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:16.598802 17283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:16.599095 17283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:16.607520 17283 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:16.607685 17283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:16.607854 17283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.134
I20240501 13:59:16.617372 17283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.134
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:16.618850 17283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:16.620662 17283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:16.633325 17288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.633932 17291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:16.634598 17289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:16.635332 17283 server_base.cc:1034] running on GCE node
I20240501 13:59:16.635962 17283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:16.636986 17283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:16.638190 17283 hybrid_clock.cc:648] HybridClock initialized: now 1714571956638142 us; error 43 us; skew 500 ppm
I20240501 13:59:16.638501 17283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:16.640810 17283 webserver.cc:460] Webserver started at http://127.16.36.134:40787/ using document root <none> and password file <none>
I20240501 13:59:16.641549 17283 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:16.641667 17283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:16.641978 17283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:16.644670 17283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "200e98827f7c44d29cd71ffd7faec1ca"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:16.645469 17283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "200e98827f7c44d29cd71ffd7faec1ca"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:16.650899 17283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 13:59:16.654572 17297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.656112 17283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:16.656275 17283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "200e98827f7c44d29cd71ffd7faec1ca"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:16.656476 17283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:16.673934 17283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:16.674902 17283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:16.675166 17283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:16.676110 17283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:16.677636 17283 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:16.677723 17283 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.677814 17283 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:16.677868 17283 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:16.709029 17283 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.134:40959
I20240501 13:59:16.709082 17409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.134:40959 every 8 connection(s)
I20240501 13:59:16.710363 17283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:16.713865 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 17283
I20240501 13:59:16.714082 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 13:59:16.717445 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.135:0
--local_ip_for_outbound_sockets=127.16.36.135
--webserver_interface=127.16.36.135
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:16.723898 17410 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:16.724265 17410 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:16.725335 17410 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:16.727030 16573 ts_manager.cc:194] Registered new tserver with Master: 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959)
I20240501 13:59:16.728283 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.134:45765
I20240501 13:59:16.774758 17150 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
W20240501 13:59:17.072109 17413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:17.072417 17413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:17.072571 17413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:17.080322 17413 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:17.080451 17413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:17.080621 17413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.135
I20240501 13:59:17.090241 17413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.135
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:17.091667 17413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:17.093475 17413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:17.106024 17418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:17.106549 17421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:17.107174 17419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:17.107611 17413 server_base.cc:1034] running on GCE node
I20240501 13:59:17.108240 17413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:17.109254 17413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:17.110476 17413 hybrid_clock.cc:648] HybridClock initialized: now 1714571957110419 us; error 43 us; skew 500 ppm
I20240501 13:59:17.110877 17413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:17.113273 17413 webserver.cc:460] Webserver started at http://127.16.36.135:39357/ using document root <none> and password file <none>
I20240501 13:59:17.114050 17413 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:17.114154 17413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:17.114493 17413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:17.117841 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "efb8dd25e14a46ae89eda6bbe5f940a2"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:17.118836 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "efb8dd25e14a46ae89eda6bbe5f940a2"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:17.124671 17413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:17.128307 17427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.129802 17413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:17.129992 17413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "efb8dd25e14a46ae89eda6bbe5f940a2"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:17.130211 17413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:17.144260 17413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:17.145153 17413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:17.145429 17413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:17.146394 17413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:17.148005 17413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:17.148106 17413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.148223 17413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:17.148295 17413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.182262 17413 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.135:39399
I20240501 13:59:17.182341 17539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.135:39399 every 8 connection(s)
I20240501 13:59:17.183873 17413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:17.188300 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 17413
I20240501 13:59:17.188524 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 13:59:17.192131 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.136:0
--local_ip_for_outbound_sockets=127.16.36.136
--webserver_interface=127.16.36.136
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43309
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:17.199282 17540 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:17.199652 17540 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:17.200649 17540 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:17.202315 16573 ts_manager.cc:194] Registered new tserver with Master: efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399)
I20240501 13:59:17.203577 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.135:43527
I20240501 13:59:17.260087 17280 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
W20240501 13:59:17.541697 17543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:17.542003 17543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:17.542148 17543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:17.549932 17543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:17.550107 17543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:17.550280 17543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.136
I20240501 13:59:17.560102 17543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.136
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:17.561529 17543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:17.563401 17543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:17.576454 17549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:17.576444 17548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:17.577291 17551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:17.578032 17543 server_base.cc:1034] running on GCE node
I20240501 13:59:17.578886 17543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:17.579974 17543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:17.581225 17543 hybrid_clock.cc:648] HybridClock initialized: now 1714571957581170 us; error 44 us; skew 500 ppm
I20240501 13:59:17.581602 17543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:17.584204 17543 webserver.cc:460] Webserver started at http://127.16.36.136:43915/ using document root <none> and password file <none>
I20240501 13:59:17.585014 17543 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:17.585153 17543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:17.585469 17543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:17.588361 17543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "2e5565b8761548539de35f9318bec337"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:17.589206 17543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "2e5565b8761548539de35f9318bec337"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:17.594767 17543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 13:59:17.598300 17557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.599869 17543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:17.600041 17543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2e5565b8761548539de35f9318bec337"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:17.600461 17543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:17.619383 17543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:17.620244 17543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:17.620491 17543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:17.621423 17543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:17.623050 17543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:17.623140 17543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.623248 17543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:17.623306 17543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:17.656445 17543 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.136:40683
I20240501 13:59:17.656508 17669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.136:40683 every 8 connection(s)
I20240501 13:59:17.657879 17543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 13:59:17.662575 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 17543
I20240501 13:59:17.662784 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 13:59:17.671432 17670 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:17.671725 17670 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:17.672516 17670 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:17.673944 16573 ts_manager.cc:194] Registered new tserver with Master: 2e5565b8761548539de35f9318bec337 (127.16.36.136:40683)
I20240501 13:59:17.674793 16573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.136:57479
I20240501 13:59:17.679579 16530 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 13:59:17.701416 16530 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 13:59:17.701599 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 17153
I20240501 13:59:17.710528 17275 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:17.731258 17410 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
I20240501 13:59:17.795289 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 17153
I20240501 13:59:17.812095 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 17283
I20240501 13:59:17.821130 17405 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:17.908923 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 17283
I20240501 13:59:17.931918 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 17413
I20240501 13:59:17.940450 17535 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:18.023262 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 17413
I20240501 13:59:18.039196 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 17543
I20240501 13:59:18.047854 17665 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:18.129384 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 17543
I20240501 13:59:21.899868 16573 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51592:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:21.940177 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 92cf36a8fccc4cae80751e98374448d8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.940176 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 92cf36a8fccc4cae80751e98374448d8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.941745 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cf36a8fccc4cae80751e98374448d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.941746 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cf36a8fccc4cae80751e98374448d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.943598 16693 tablet_service.cc:1467] Processing CreateTablet for tablet ac5a6954888149209b7b2fed2ab849f2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.943619 16692 tablet_service.cc:1467] Processing CreateTablet for tablet a25c0fb5b5264854a229ed8acdbed791 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.944137 16693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5a6954888149209b7b2fed2ab849f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.944401 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25c0fb5b5264854a229ed8acdbed791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.945003 16824 tablet_service.cc:1467] Processing CreateTablet for tablet ac5a6954888149209b7b2fed2ab849f2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.945488 16824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5a6954888149209b7b2fed2ab849f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.947700 17084 tablet_service.cc:1467] Processing CreateTablet for tablet a25c0fb5b5264854a229ed8acdbed791 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.949213 17084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25c0fb5b5264854a229ed8acdbed791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.951165 17085 tablet_service.cc:1467] Processing CreateTablet for tablet 92cf36a8fccc4cae80751e98374448d8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.953526 17085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92cf36a8fccc4cae80751e98374448d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.951143 16955 tablet_service.cc:1467] Processing CreateTablet for tablet ac5a6954888149209b7b2fed2ab849f2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.954736 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5a6954888149209b7b2fed2ab849f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.950531 16954 tablet_service.cc:1467] Processing CreateTablet for tablet a25c0fb5b5264854a229ed8acdbed791 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5585183c800b40799c84cddf775d90e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:21.960916 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25c0fb5b5264854a229ed8acdbed791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:21.983374 17701 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:21.986691 17702 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:21.986721 17700 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:21.990749 17702 tablet_bootstrap.cc:654] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.991710 17700 tablet_bootstrap.cc:654] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.991701 17701 tablet_bootstrap.cc:654] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.991720 17699 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:21.992511 17702 log.cc:824] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:21.992970 17701 log.cc:824] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:21.992970 17700 log.cc:824] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:21.995821 17699 tablet_bootstrap.cc:654] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:21.997180 17699 log.cc:824] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:21.998406 17702 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:21.999166 17702 ts_tablet_manager.cc:1367] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240501 13:59:22.008687 17700 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:22.009341 17700 ts_tablet_manager.cc:1367] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.013s
I20240501 13:59:22.012652 17701 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:22.012979 17701 ts_tablet_manager.cc:1367] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.012s
I20240501 13:59:22.013096 17699 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:22.013432 17699 ts_tablet_manager.cc:1367] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.013s
I20240501 13:59:22.013490 17702 raft_consensus.cc:348] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.014034 17702 raft_consensus.cc:374] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.014178 17702 raft_consensus.cc:729] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.014988 17702 consensus_queue.cc:260] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.019495 17700 raft_consensus.cc:348] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.020108 17700 raft_consensus.cc:374] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.020223 17700 raft_consensus.cc:729] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.021061 17700 consensus_queue.cc:260] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.023233 17699 raft_consensus.cc:348] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.023648 17702 ts_tablet_manager.cc:1398] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.024s user 0.013s sys 0.009s
I20240501 13:59:22.023916 17699 raft_consensus.cc:374] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.024096 17699 raft_consensus.cc:729] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.024243 17702 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:22.025014 17699 consensus_queue.cc:260] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.025082 17701 raft_consensus.cc:348] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.025538 17701 raft_consensus.cc:374] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.025642 17701 raft_consensus.cc:729] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.026273 17702 tablet_bootstrap.cc:654] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:22.026297 17701 consensus_queue.cc:260] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.030468 17699 ts_tablet_manager.cc:1398] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20240501 13:59:22.031057 17699 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
W20240501 13:59:22.031533 17021 tablet.cc:2369] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:22.033027 17699 tablet_bootstrap.cc:654] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:22.038542 17701 ts_tablet_manager.cc:1398] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.025s user 0.008s sys 0.014s
I20240501 13:59:22.038828 17702 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:22.039026 17699 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:22.039027 17702 ts_tablet_manager.cc:1367] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20240501 13:59:22.039239 17699 ts_tablet_manager.cc:1367] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 13:59:22.039417 17700 ts_tablet_manager.cc:1398] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.030s user 0.013s sys 0.000s
I20240501 13:59:22.040145 17702 raft_consensus.cc:348] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.040391 17702 raft_consensus.cc:374] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.040483 17702 raft_consensus.cc:729] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.040831 17702 consensus_queue.cc:260] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.041237 17700 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:22.041288 17701 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:22.041410 17702 ts_tablet_manager.cc:1398] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:22.042968 17700 tablet_bootstrap.cc:654] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:22.042964 17701 tablet_bootstrap.cc:654] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:22.040764 17699 raft_consensus.cc:348] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.044768 17699 raft_consensus.cc:374] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.044941 17699 raft_consensus.cc:729] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.045325 17699 consensus_queue.cc:260] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.046082 17699 ts_tablet_manager.cc:1398] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 13:59:22.046350 17699 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:22.046497 17700 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:22.046705 17700 ts_tablet_manager.cc:1367] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 13:59:22.047785 17699 tablet_bootstrap.cc:654] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:22.047822 17700 raft_consensus.cc:348] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.048038 17700 raft_consensus.cc:374] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.048137 17700 raft_consensus.cc:729] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.048364 17700 consensus_queue.cc:260] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.048961 17700 ts_tablet_manager.cc:1398] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:22.050076 17701 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:22.050274 17701 ts_tablet_manager.cc:1367] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240501 13:59:22.051409 17699 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:22.051466 17701 raft_consensus.cc:348] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.051632 17699 ts_tablet_manager.cc:1367] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:22.051770 17701 raft_consensus.cc:374] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.051887 17701 raft_consensus.cc:729] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.052158 17701 consensus_queue.cc:260] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.052346 17699 raft_consensus.cc:348] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.052609 17699 raft_consensus.cc:374] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.052700 17699 raft_consensus.cc:729] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.052767 17701 ts_tablet_manager.cc:1398] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 13:59:22.052956 17699 consensus_queue.cc:260] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.054013 17699 ts_tablet_manager.cc:1398] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:22.072899 17708 raft_consensus.cc:482] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:22.073184 17708 raft_consensus.cc:504] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
W20240501 13:59:22.074139 16891 tablet.cc:2369] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:22.075124 17708 leader_election.cc:288] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:22.083199 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25c0fb5b5264854a229ed8acdbed791" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 13:59:22.083433 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25c0fb5b5264854a229ed8acdbed791" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f" is_pre_election: true
I20240501 13:59:22.083884 16975 raft_consensus.cc:2454] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 0.
I20240501 13:59:22.084002 16714 raft_consensus.cc:2454] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 0.
I20240501 13:59:22.084632 17041 leader_election.cc:302] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:22.085080 17708 raft_consensus.cc:2781] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:22.085275 17708 raft_consensus.cc:482] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:22.085386 17708 raft_consensus.cc:3037] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.087718 17708 raft_consensus.cc:504] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.088405 17708 leader_election.cc:288] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 election: Requested vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:22.088997 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25c0fb5b5264854a229ed8acdbed791" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f"
I20240501 13:59:22.088997 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25c0fb5b5264854a229ed8acdbed791" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:22.089231 16714 raft_consensus.cc:3037] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.089231 16975 raft_consensus.cc:3037] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.092545 16975 raft_consensus.cc:2454] T a25c0fb5b5264854a229ed8acdbed791 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 1.
I20240501 13:59:22.092545 16714 raft_consensus.cc:2454] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 1.
I20240501 13:59:22.093122 17041 leader_election.cc:302] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:22.093500 17708 raft_consensus.cc:2781] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:22.093999 17708 raft_consensus.cc:686] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Becoming Leader. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Running, Role: LEADER
I20240501 13:59:22.094550 17708 consensus_queue.cc:237] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.099565 16572 catalog_manager.cc:5484] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132). New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:22.110607 16760 tablet.cc:2369] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:22.171417 17707 raft_consensus.cc:482] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:22.171648 17707 raft_consensus.cc:504] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.173250 17707 leader_election.cc:288] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:22.180034 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5a6954888149209b7b2fed2ab849f2" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f" is_pre_election: true
I20240501 13:59:22.180302 16714 raft_consensus.cc:2454] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:22.180235 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5a6954888149209b7b2fed2ab849f2" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:22.180774 16845 raft_consensus.cc:2454] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:22.180773 16908 leader_election.cc:302] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:22.181221 17707 raft_consensus.cc:2781] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:22.181354 17707 raft_consensus.cc:482] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:22.181458 17707 raft_consensus.cc:3037] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.184060 17707 raft_consensus.cc:504] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.184759 17707 leader_election.cc:288] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Requested vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:22.185235 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5a6954888149209b7b2fed2ab849f2" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:22.185323 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5a6954888149209b7b2fed2ab849f2" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f"
I20240501 13:59:22.185461 16845 raft_consensus.cc:3037] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.185585 16714 raft_consensus.cc:3037] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.188521 16714 raft_consensus.cc:2454] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:22.188539 16845 raft_consensus.cc:2454] T ac5a6954888149209b7b2fed2ab849f2 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:22.189023 16911 leader_election.cc:302] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:22.189440 17707 raft_consensus.cc:2781] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:22.189901 17707 raft_consensus.cc:686] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Becoming Leader. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:22.190441 17707 consensus_queue.cc:237] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:22.195444 16573 catalog_manager.cc:5484] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4b54fe4701746c0930c9b0667c90519 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:22.260221 17151 tablet.cc:2369] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:22.303411 17710 raft_consensus.cc:482] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:22.303627 17710 raft_consensus.cc:504] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.305245 17710 leader_election.cc:288] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:22.313079 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cf36a8fccc4cae80751e98374448d8" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f" is_pre_election: true
I20240501 13:59:22.313359 16714 raft_consensus.cc:2454] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 0.
I20240501 13:59:22.313258 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cf36a8fccc4cae80751e98374448d8" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:22.313751 17105 raft_consensus.cc:2454] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 0.
I20240501 13:59:22.313912 16778 leader_election.cc:302] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:22.314348 17710 raft_consensus.cc:2781] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:22.314465 17710 raft_consensus.cc:482] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:22.314566 17710 raft_consensus.cc:3037] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.316604 17710 raft_consensus.cc:504] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.317230 17710 leader_election.cc:288] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 election: Requested vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:22.317965 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cf36a8fccc4cae80751e98374448d8" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:22.317965 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cf36a8fccc4cae80751e98374448d8" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f"
I20240501 13:59:22.318213 16714 raft_consensus.cc:3037] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.318213 17105 raft_consensus.cc:3037] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.321390 16714 raft_consensus.cc:2454] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:22.321390 17105 raft_consensus.cc:2454] T 92cf36a8fccc4cae80751e98374448d8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:22.321887 16778 leader_election.cc:302] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:22.322258 17710 raft_consensus.cc:2781] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:22.322896 17710 raft_consensus.cc:686] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Becoming Leader. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:22.323493 17710 consensus_queue.cc:237] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:22.328408 16573 catalog_manager.cc:5484] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: term changed from 0 to 1, leader changed from <none> to d871186c2b234914ab63dc5e136f0f03 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:22.556345 17708 consensus_queue.cc:1035] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.564338 17708 consensus_queue.cc:1035] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.679915 17707 consensus_queue.cc:1035] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.686784 17722 consensus_queue.cc:1035] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.726035 17710 consensus_queue.cc:1035] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:22.732501 17710 consensus_queue.cc:1035] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:22.775549 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:22.775938 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:22.784322 17728 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 239.987 ms
time per row: 0.119994 ms
I20240501 13:59:23.177309 16572 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51592:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:23.186792 16824 tablet_service.cc:1467] Processing CreateTablet for tablet 221688d4e2374a42b03e96d89166e0ee (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.187119 16954 tablet_service.cc:1467] Processing CreateTablet for tablet 221688d4e2374a42b03e96d89166e0ee (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.187325 16824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221688d4e2374a42b03e96d89166e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.187552 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221688d4e2374a42b03e96d89166e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.187587 17085 tablet_service.cc:1467] Processing CreateTablet for tablet 221688d4e2374a42b03e96d89166e0ee (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.188047 17085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221688d4e2374a42b03e96d89166e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.191030 16825 tablet_service.cc:1467] Processing CreateTablet for tablet a3bfe952008f438c887a7d9574fc8bb9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.191501 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfe952008f438c887a7d9574fc8bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.191422 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 3068f422a4d54d998715bffa2667854e (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.191633 17084 tablet_service.cc:1467] Processing CreateTablet for tablet a3bfe952008f438c887a7d9574fc8bb9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.191839 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3068f422a4d54d998715bffa2667854e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.192068 17084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfe952008f438c887a7d9574fc8bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.195613 16955 tablet_service.cc:1467] Processing CreateTablet for tablet a3bfe952008f438c887a7d9574fc8bb9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.196303 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfe952008f438c887a7d9574fc8bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.196362 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 3068f422a4d54d998715bffa2667854e (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.196785 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3068f422a4d54d998715bffa2667854e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.197592 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 3068f422a4d54d998715bffa2667854e (DEFAULT_TABLE table=rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:23.198246 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3068f422a4d54d998715bffa2667854e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.209268 17792 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:23.212105 17792 tablet_bootstrap.cc:654] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.215394 17790 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:23.218125 17790 tablet_bootstrap.cc:654] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.220971 17792 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:23.221232 17792 ts_tablet_manager.cc:1367] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240501 13:59:23.221428 17791 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:23.222322 17792 raft_consensus.cc:348] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.222694 17792 raft_consensus.cc:374] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.222852 17792 raft_consensus.cc:729] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.223058 17791 tablet_bootstrap.cc:654] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.223181 17792 consensus_queue.cc:260] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.223920 17792 ts_tablet_manager.cc:1398] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:23.229570 17792 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:23.231297 17792 tablet_bootstrap.cc:654] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.231503 17793 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:23.232247 17791 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:23.232513 17791 ts_tablet_manager.cc:1367] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 13:59:23.233728 17791 raft_consensus.cc:348] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.234127 17791 raft_consensus.cc:374] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.234295 17791 raft_consensus.cc:729] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.234735 17793 tablet_bootstrap.cc:654] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.234727 17791 consensus_queue.cc:260] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.236115 17790 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:23.236279 17790 ts_tablet_manager.cc:1367] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.011s
I20240501 13:59:23.237442 17790 raft_consensus.cc:348] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.237708 17790 raft_consensus.cc:374] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.237819 17790 raft_consensus.cc:729] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.238121 17790 consensus_queue.cc:260] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.238809 17790 ts_tablet_manager.cc:1398] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 13:59:23.239212 17790 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:23.239910 17792 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:23.240105 17792 ts_tablet_manager.cc:1367] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 13:59:23.240967 17792 raft_consensus.cc:348] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.241220 17792 raft_consensus.cc:374] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.241315 17792 raft_consensus.cc:729] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.241766 17790 tablet_bootstrap.cc:654] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.241688 17792 consensus_queue.cc:260] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.242451 17792 ts_tablet_manager.cc:1398] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:23.243582 17791 ts_tablet_manager.cc:1398] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240501 13:59:23.245481 17793 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:23.245744 17793 ts_tablet_manager.cc:1367] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20240501 13:59:23.246287 17790 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:23.246465 17790 ts_tablet_manager.cc:1367] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 13:59:23.246892 17793 raft_consensus.cc:348] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.247179 17793 raft_consensus.cc:374] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.247289 17793 raft_consensus.cc:729] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.247598 17793 consensus_queue.cc:260] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.248183 17793 ts_tablet_manager.cc:1398] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 13:59:23.247524 17790 raft_consensus.cc:348] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.248647 17790 raft_consensus.cc:374] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.248677 17793 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:23.248733 17790 raft_consensus.cc:729] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.249009 17790 consensus_queue.cc:260] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.249639 17790 ts_tablet_manager.cc:1398] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:23.250267 17790 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:23.251492 17793 tablet_bootstrap.cc:654] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.252030 17790 tablet_bootstrap.cc:654] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.254096 17790 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:23.254233 17790 ts_tablet_manager.cc:1367] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:23.254887 17790 raft_consensus.cc:348] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.255088 17790 raft_consensus.cc:374] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.255187 17790 raft_consensus.cc:729] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.255371 17793 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:23.255534 17793 ts_tablet_manager.cc:1367] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 13:59:23.255421 17790 consensus_queue.cc:260] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.255964 17790 ts_tablet_manager.cc:1398] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:23.256294 17793 raft_consensus.cc:348] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.256466 17793 raft_consensus.cc:374] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.256542 17793 raft_consensus.cc:729] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.256796 17793 consensus_queue.cc:260] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.257254 17793 ts_tablet_manager.cc:1398] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:23.257557 17793 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:23.259002 17793 tablet_bootstrap.cc:654] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.263756 17793 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:23.263921 17793 ts_tablet_manager.cc:1367] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 13:59:23.264544 17793 raft_consensus.cc:348] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.264730 17793 raft_consensus.cc:374] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.264832 17793 raft_consensus.cc:729] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.265051 17793 consensus_queue.cc:260] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.265524 17793 ts_tablet_manager.cc:1398] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 13:59:23.303119 17783 raft_consensus.cc:482] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:23.303320 17783 raft_consensus.cc:504] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.304232 17783 leader_election.cc:288] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:23.304930 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:23.305301 16845 raft_consensus.cc:2454] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:23.305867 16911 leader_election.cc:302] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:23.306185 17783 raft_consensus.cc:2781] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:23.306329 17783 raft_consensus.cc:482] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:23.306438 17783 raft_consensus.cc:3037] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.308921 17783 raft_consensus.cc:504] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.309983 17783 leader_election.cc:288] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Requested vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:23.310137 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:23.310384 16845 raft_consensus.cc:3037] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.311889 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:23.311890 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:23.312206 17105 raft_consensus.cc:2454] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:23.312992 16845 raft_consensus.cc:2454] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:23.313383 16911 leader_election.cc:302] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters: 70ff9aaa9fa2414ea55225aafbd8fd89
I20240501 13:59:23.313719 17783 raft_consensus.cc:2781] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:23.313859 17783 raft_consensus.cc:686] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Becoming Leader. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:23.314122 17783 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.316854 16569 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4b54fe4701746c0930c9b0667c90519 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:23.392443 17718 raft_consensus.cc:482] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:23.392665 17718 raft_consensus.cc:504] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.393497 17718 leader_election.cc:288] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:23.394121 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 13:59:23.394450 16975 raft_consensus.cc:2454] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 0.
I20240501 13:59:23.394975 17041 leader_election.cc:302] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:23.395339 17718 raft_consensus.cc:2781] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:23.395452 17718 raft_consensus.cc:482] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:23.395557 17718 raft_consensus.cc:3037] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.397548 17718 raft_consensus.cc:504] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.398221 17718 leader_election.cc:288] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 election: Requested vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:23.398828 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:23.399068 16975 raft_consensus.cc:3037] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.400703 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:23.401024 16845 raft_consensus.cc:2454] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 0.
I20240501 13:59:23.400931 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:23.401208 16844 raft_consensus.cc:3037] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.401896 16975 raft_consensus.cc:2454] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 1.
I20240501 13:59:23.402382 17041 leader_election.cc:302] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:23.402719 17718 raft_consensus.cc:2781] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:23.402860 17718 raft_consensus.cc:686] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Becoming Leader. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Running, Role: LEADER
I20240501 13:59:23.403153 17718 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:23.403617 16844 raft_consensus.cc:2454] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 1.
I20240501 13:59:23.406350 16569 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132). New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:23.423651 17798 raft_consensus.cc:482] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:23.423877 17798 raft_consensus.cc:504] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.425472 17798 leader_election.cc:288] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:23.432737 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3068f422a4d54d998715bffa2667854e" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 13:59:23.433030 16975 raft_consensus.cc:2454] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:23.433013 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3068f422a4d54d998715bffa2667854e" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:23.433265 16844 raft_consensus.cc:2454] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:23.433511 16650 leader_election.cc:302] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:23.433910 17798 raft_consensus.cc:2781] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:23.434036 17798 raft_consensus.cc:482] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:23.434140 17798 raft_consensus.cc:3037] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.436151 17798 raft_consensus.cc:504] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.436730 17798 leader_election.cc:288] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Requested vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:23.437207 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3068f422a4d54d998715bffa2667854e" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:23.437242 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3068f422a4d54d998715bffa2667854e" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:23.437400 16975 raft_consensus.cc:3037] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.437438 16844 raft_consensus.cc:3037] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.439318 16975 raft_consensus.cc:2454] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:23.439361 16844 raft_consensus.cc:2454] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:23.439713 16650 leader_election.cc:302] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:23.440001 17798 raft_consensus.cc:2781] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:23.440465 17798 raft_consensus.cc:686] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 1 LEADER]: Becoming Leader. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Running, Role: LEADER
I20240501 13:59:23.440909 17798 consensus_queue.cc:237] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:23.444325 16569 catalog_manager.cc:5484] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f8b1516d5142709756d06b06c5d55f (127.16.36.129). New cstate: current_term: 1 leader_uuid: "10f8b1516d5142709756d06b06c5d55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:23.802417 17105 raft_consensus.cc:3037] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:59:23.803939 17812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:23.804384 17812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 13:59:23.806329 17783 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:23.815814 17783 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:23.817454 17812 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 13:59:23.892131 17718 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:23.994097 17846 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240501 13:59:24.001479 17845 consensus_queue.cc:1035] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:24.003389 17843 consensus_queue.cc:1035] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:24.140077 16908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:37337, user_credentials={real_user=slave}} blocked reactor thread for 63995.7us
INSERT report
rows total: 2000
time total: 332.818 ms
time per row: 0.166409 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240501 13:59:25.253043 16573 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51592:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:25.263021 16694 tablet_service.cc:1467] Processing CreateTablet for tablet f9c8c82da2da4173899f5b9e03894da8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.263077 17084 tablet_service.cc:1467] Processing CreateTablet for tablet f9c8c82da2da4173899f5b9e03894da8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.263518 17084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c8c82da2da4173899f5b9e03894da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.263541 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c8c82da2da4173899f5b9e03894da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.264328 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 8f27c70cad684568af8d8119559475cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.264367 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 35a493549e9a4975a3820fabb6e6e491 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.264775 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a493549e9a4975a3820fabb6e6e491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.264784 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27c70cad684568af8d8119559475cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.267189 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 35a493549e9a4975a3820fabb6e6e491 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.267643 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a493549e9a4975a3820fabb6e6e491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.267880 17085 tablet_service.cc:1467] Processing CreateTablet for tablet 35a493549e9a4975a3820fabb6e6e491 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.268287 17085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a493549e9a4975a3820fabb6e6e491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.270810 16823 tablet_service.cc:1467] Processing CreateTablet for tablet f9c8c82da2da4173899f5b9e03894da8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.271265 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c8c82da2da4173899f5b9e03894da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.279994 17885 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:25.279976 17083 tablet_service.cc:1467] Processing CreateTablet for tablet 8f27c70cad684568af8d8119559475cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.280517 17083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27c70cad684568af8d8119559475cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.282049 17885 tablet_bootstrap.cc:654] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.283151 16693 tablet_service.cc:1467] Processing CreateTablet for tablet 8f27c70cad684568af8d8119559475cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=ad9da273de614774a3d32b697dad72cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.283610 16693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f27c70cad684568af8d8119559475cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.290908 17888 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:25.292790 17888 tablet_bootstrap.cc:654] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.294214 17886 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:25.296455 17886 tablet_bootstrap.cc:654] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.297201 17888 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:25.297199 17885 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:25.297365 17885 ts_tablet_manager.cc:1367] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20240501 13:59:25.297503 17888 ts_tablet_manager.cc:1367] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 13:59:25.297508 17890 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:25.298243 17885 raft_consensus.cc:348] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.298489 17885 raft_consensus.cc:374] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.298578 17885 raft_consensus.cc:729] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.298334 17888 raft_consensus.cc:348] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.298902 17888 raft_consensus.cc:374] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.298858 17885 consensus_queue.cc:260] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.299036 17888 raft_consensus.cc:729] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.299453 17885 ts_tablet_manager.cc:1398] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:25.299430 17888 consensus_queue.cc:260] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.299877 17885 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:25.300055 17890 tablet_bootstrap.cc:654] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.300160 17888 ts_tablet_manager.cc:1398] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:25.300767 17888 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:25.301687 17885 tablet_bootstrap.cc:654] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.302556 17888 tablet_bootstrap.cc:654] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.309574 17888 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:25.309967 17888 ts_tablet_manager.cc:1367] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240501 13:59:25.311249 17888 raft_consensus.cc:348] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.311695 17888 raft_consensus.cc:374] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.311946 17888 raft_consensus.cc:729] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.312616 17888 consensus_queue.cc:260] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.314206 17888 ts_tablet_manager.cc:1398] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 13:59:25.323328 17888 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:25.325906 17888 tablet_bootstrap.cc:654] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.336694 17886 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:25.337093 17886 ts_tablet_manager.cc:1367] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.015s
I20240501 13:59:25.338338 17886 raft_consensus.cc:348] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.339849 17888 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:25.340173 17888 ts_tablet_manager.cc:1367] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20240501 13:59:25.341370 17888 raft_consensus.cc:348] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.341786 17888 raft_consensus.cc:374] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.341981 17888 raft_consensus.cc:729] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.342761 17886 raft_consensus.cc:374] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.342444 17888 consensus_queue.cc:260] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.343221 17886 raft_consensus.cc:729] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.344131 17888 ts_tablet_manager.cc:1398] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:25.344652 17886 consensus_queue.cc:260] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.347491 17886 ts_tablet_manager.cc:1398] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240501 13:59:25.348013 17886 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:25.350531 17886 tablet_bootstrap.cc:654] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.359705 17886 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:25.360121 17886 ts_tablet_manager.cc:1367] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.010s
I20240501 13:59:25.361305 17886 raft_consensus.cc:348] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.361789 17886 raft_consensus.cc:374] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.361963 17886 raft_consensus.cc:729] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.362382 17886 consensus_queue.cc:260] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.363672 17886 ts_tablet_manager.cc:1398] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:25.368250 17885 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:25.368484 17885 ts_tablet_manager.cc:1367] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.069s user 0.020s sys 0.045s
I20240501 13:59:25.369413 17885 raft_consensus.cc:348] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.369707 17885 raft_consensus.cc:374] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.369767 17890 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:25.369820 17885 raft_consensus.cc:729] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.370038 17890 ts_tablet_manager.cc:1367] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.073s user 0.034s sys 0.034s
I20240501 13:59:25.370091 17885 consensus_queue.cc:260] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.370735 17885 ts_tablet_manager.cc:1398] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:25.371117 17885 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:25.371124 17890 raft_consensus.cc:348] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.371436 17890 raft_consensus.cc:374] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.371572 17890 raft_consensus.cc:729] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.371892 17890 consensus_queue.cc:260] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.372619 17890 ts_tablet_manager.cc:1398] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:25.372925 17885 tablet_bootstrap.cc:654] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.377302 17885 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:25.377497 17885 ts_tablet_manager.cc:1367] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 13:59:25.378505 17885 raft_consensus.cc:348] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.378770 17885 raft_consensus.cc:374] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.378873 17885 raft_consensus.cc:729] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.379118 17885 consensus_queue.cc:260] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.379724 17885 ts_tablet_manager.cc:1398] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:25.486776 17865 raft_consensus.cc:482] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:25.487025 17865 raft_consensus.cc:504] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.487937 17865 leader_election.cc:288] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:25.488569 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c8c82da2da4173899f5b9e03894da8" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:25.488821 16844 raft_consensus.cc:2454] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:25.489312 16650 leader_election.cc:302] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:25.489647 17865 raft_consensus.cc:2781] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:25.489806 17865 raft_consensus.cc:482] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:25.489923 17865 raft_consensus.cc:3037] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.492676 17865 raft_consensus.cc:504] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.493330 17865 leader_election.cc:288] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Requested vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:25.493912 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c8c82da2da4173899f5b9e03894da8" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:25.494168 16844 raft_consensus.cc:3037] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.495522 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c8c82da2da4173899f5b9e03894da8" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:25.495460 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c8c82da2da4173899f5b9e03894da8" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:25.495743 17104 raft_consensus.cc:3037] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.496524 16844 raft_consensus.cc:2454] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:25.496960 16650 leader_election.cc:302] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:25.497287 17865 raft_consensus.cc:2781] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:25.497777 17865 raft_consensus.cc:686] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 1 LEADER]: Becoming Leader. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Running, Role: LEADER
I20240501 13:59:25.498112 17865 consensus_queue.cc:237] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:25.498490 17104 raft_consensus.cc:2454] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:25.501508 16569 catalog_manager.cc:5484] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f8b1516d5142709756d06b06c5d55f (127.16.36.129). New cstate: current_term: 1 leader_uuid: "10f8b1516d5142709756d06b06c5d55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:25.545114 17861 raft_consensus.cc:482] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:25.545307 17861 raft_consensus.cc:504] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.546070 17861 leader_election.cc:288] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:25.546587 16713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a493549e9a4975a3820fabb6e6e491" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f" is_pre_election: true
I20240501 13:59:25.546713 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a493549e9a4975a3820fabb6e6e491" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:25.546878 16713 raft_consensus.cc:2454] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:25.546959 17104 raft_consensus.cc:2454] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:25.547389 16908 leader_election.cc:302] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:25.547694 17861 raft_consensus.cc:2781] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:25.547802 17861 raft_consensus.cc:482] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:25.547900 17861 raft_consensus.cc:3037] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.549791 17861 raft_consensus.cc:504] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.550374 17861 leader_election.cc:288] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Requested vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:25.550932 16713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a493549e9a4975a3820fabb6e6e491" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f"
I20240501 13:59:25.551085 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a493549e9a4975a3820fabb6e6e491" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:25.551169 16713 raft_consensus.cc:3037] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.551306 17104 raft_consensus.cc:3037] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.553754 16713 raft_consensus.cc:2454] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:25.553756 17104 raft_consensus.cc:2454] T 35a493549e9a4975a3820fabb6e6e491 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:25.554165 16908 leader_election.cc:302] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:25.554459 17861 raft_consensus.cc:2781] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:25.554881 17861 raft_consensus.cc:686] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Becoming Leader. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:25.555372 17861 consensus_queue.cc:237] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:25.558456 16569 catalog_manager.cc:5484] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4b54fe4701746c0930c9b0667c90519 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:25.600898 17865 raft_consensus.cc:482] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:25.601161 17865 raft_consensus.cc:504] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.601894 17865 leader_election.cc:288] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:25.602375 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27c70cad684568af8d8119559475cc" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:25.602401 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27c70cad684568af8d8119559475cc" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:25.602665 16844 raft_consensus.cc:2454] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:25.602715 17104 raft_consensus.cc:2454] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:25.603147 16650 leader_election.cc:302] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:25.603441 17865 raft_consensus.cc:2781] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:25.603526 17865 raft_consensus.cc:482] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:25.603605 17865 raft_consensus.cc:3037] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.605831 17865 raft_consensus.cc:504] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.606513 17865 leader_election.cc:288] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Requested vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:25.606940 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27c70cad684568af8d8119559475cc" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:25.606936 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27c70cad684568af8d8119559475cc" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:25.607123 16844 raft_consensus.cc:3037] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.607142 17104 raft_consensus.cc:3037] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.609171 16844 raft_consensus.cc:2454] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:25.609194 17104 raft_consensus.cc:2454] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:25.609597 16650 leader_election.cc:302] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:25.609902 17865 raft_consensus.cc:2781] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:25.610069 17865 raft_consensus.cc:686] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 1 LEADER]: Becoming Leader. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Running, Role: LEADER
I20240501 13:59:25.610324 17865 consensus_queue.cc:237] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:25.613205 16569 catalog_manager.cc:5484] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f8b1516d5142709756d06b06c5d55f (127.16.36.129). New cstate: current_term: 1 leader_uuid: "10f8b1516d5142709756d06b06c5d55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:25.913566 17865 consensus_queue.cc:1035] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:25.920534 17865 consensus_queue.cc:1035] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:26.038540 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:26.039002 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 13:59:26.044937 17900 consensus_queue.cc:1035] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:26.047860 17905 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 13:59:26.052491 17865 consensus_queue.cc:1035] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 13:59:26.118584 17861 consensus_queue.cc:1035] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:26.123888 17861 consensus_queue.cc:1035] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 299.158 ms
time per row: 0.149579 ms
I20240501 13:59:26.507148 16573 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51592:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:26.517247 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 101e3579d2954f2aac17d0f10ac84865 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.517248 16693 tablet_service.cc:1467] Processing CreateTablet for tablet 101e3579d2954f2aac17d0f10ac84865 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.517793 16693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101e3579d2954f2aac17d0f10ac84865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.517805 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101e3579d2954f2aac17d0f10ac84865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.518080 16955 tablet_service.cc:1467] Processing CreateTablet for tablet 5f408fdebfb84082b0a55ad059e90694 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.518443 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f408fdebfb84082b0a55ad059e90694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.519217 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 5f408fdebfb84082b0a55ad059e90694 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.519624 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f408fdebfb84082b0a55ad059e90694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.522096 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 101e3579d2954f2aac17d0f10ac84865 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.522513 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101e3579d2954f2aac17d0f10ac84865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.523113 17085 tablet_service.cc:1467] Processing CreateTablet for tablet 5f408fdebfb84082b0a55ad059e90694 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.523523 17085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f408fdebfb84082b0a55ad059e90694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.528038 16954 tablet_service.cc:1467] Processing CreateTablet for tablet 2269a465c771402d882a363a579ba508 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.528537 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2269a465c771402d882a363a579ba508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.529582 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 2269a465c771402d882a363a579ba508 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.530148 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2269a465c771402d882a363a579ba508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.531457 17083 tablet_service.cc:1467] Processing CreateTablet for tablet 2269a465c771402d882a363a579ba508 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:26.531949 17083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2269a465c771402d882a363a579ba508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:26.532541 17971 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:26.535055 17971 tablet_bootstrap.cc:654] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.552757 17974 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:26.553314 17972 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:26.554124 17970 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:26.554322 17971 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:26.554834 17971 ts_tablet_manager.cc:1367] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240501 13:59:26.554924 17974 tablet_bootstrap.cc:654] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.555686 17972 tablet_bootstrap.cc:654] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.556025 17971 raft_consensus.cc:348] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.556339 17971 raft_consensus.cc:374] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.556483 17971 raft_consensus.cc:729] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.556885 17971 consensus_queue.cc:260] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.558437 17970 tablet_bootstrap.cc:654] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.559284 17971 ts_tablet_manager.cc:1398] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:26.559686 17971 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:26.560986 17972 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:26.561228 17972 ts_tablet_manager.cc:1367] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 13:59:26.562197 17971 tablet_bootstrap.cc:654] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.562171 17972 raft_consensus.cc:348] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.562539 17972 raft_consensus.cc:374] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.562700 17972 raft_consensus.cc:729] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.563066 17972 consensus_queue.cc:260] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.564052 17972 ts_tablet_manager.cc:1398] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 13:59:26.564446 17972 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:26.566941 17972 tablet_bootstrap.cc:654] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.568522 17974 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:26.569558 17974 ts_tablet_manager.cc:1367] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240501 13:59:26.571331 17970 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:26.571543 17970 ts_tablet_manager.cc:1367] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240501 13:59:26.571688 17971 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:26.571924 17971 ts_tablet_manager.cc:1367] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240501 13:59:26.572485 17974 raft_consensus.cc:348] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.572695 17970 raft_consensus.cc:348] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.572968 17970 raft_consensus.cc:374] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.573082 17970 raft_consensus.cc:729] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.572999 17971 raft_consensus.cc:348] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.573315 17971 raft_consensus.cc:374] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.573418 17971 raft_consensus.cc:729] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.573379 17970 consensus_queue.cc:260] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.573851 17971 consensus_queue.cc:260] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.574178 17970 ts_tablet_manager.cc:1398] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:26.574509 17970 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:26.575471 17971 ts_tablet_manager.cc:1398] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 13:59:26.572836 17974 raft_consensus.cc:374] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.577268 17970 tablet_bootstrap.cc:654] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.577877 17972 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:26.577201 17974 raft_consensus.cc:729] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.578090 17972 ts_tablet_manager.cc:1367] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240501 13:59:26.578989 17974 consensus_queue.cc:260] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.579172 17972 raft_consensus.cc:348] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.579494 17972 raft_consensus.cc:374] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.579599 17972 raft_consensus.cc:729] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.580176 17972 consensus_queue.cc:260] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.580951 17972 ts_tablet_manager.cc:1398] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:26.581666 17972 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:26.581988 17974 ts_tablet_manager.cc:1398] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240501 13:59:26.582394 17974 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:26.584206 17972 tablet_bootstrap.cc:654] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.585916 17974 tablet_bootstrap.cc:654] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:26.588991 17972 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:26.589206 17972 ts_tablet_manager.cc:1367] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240501 13:59:26.590428 17972 raft_consensus.cc:348] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.590744 17972 raft_consensus.cc:374] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.590848 17972 raft_consensus.cc:729] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.591142 17972 consensus_queue.cc:260] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.591817 17972 ts_tablet_manager.cc:1398] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:26.591909 17974 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:26.592106 17974 ts_tablet_manager.cc:1367] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20240501 13:59:26.592528 17970 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:26.592713 17970 ts_tablet_manager.cc:1367] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240501 13:59:26.592934 17974 raft_consensus.cc:348] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.593189 17974 raft_consensus.cc:374] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.593269 17974 raft_consensus.cc:729] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.593401 17970 raft_consensus.cc:348] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.593642 17970 raft_consensus.cc:374] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:26.593546 17974 consensus_queue.cc:260] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.593724 17970 raft_consensus.cc:729] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:26.594112 17974 ts_tablet_manager.cc:1398] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:26.594029 17970 consensus_queue.cc:260] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.594924 17970 ts_tablet_manager.cc:1398] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 13:59:26.600423 17726 raft_consensus.cc:482] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:26.600616 17726 raft_consensus.cc:504] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.601528 17726 leader_election.cc:288] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:26.601981 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:26.602243 17104 raft_consensus.cc:2454] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 0.
I20240501 13:59:26.602764 16778 leader_election.cc:302] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:26.603081 17726 raft_consensus.cc:2781] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:26.603224 17726 raft_consensus.cc:482] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:26.603346 17726 raft_consensus.cc:3037] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.606756 17726 raft_consensus.cc:504] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.607650 17726 leader_election.cc:288] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 election: Requested vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:26.608223 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:26.608451 17104 raft_consensus.cc:3037] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.610488 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 13:59:26.610669 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:26.610810 16974 raft_consensus.cc:2454] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 0.
I20240501 13:59:26.611219 17104 raft_consensus.cc:2454] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:26.611671 16778 leader_election.cc:302] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters: f4b54fe4701746c0930c9b0667c90519
I20240501 13:59:26.611968 17726 raft_consensus.cc:2781] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:26.612380 17726 raft_consensus.cc:686] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Becoming Leader. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:26.612686 17726 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.615639 16573 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: term changed from 0 to 1, leader changed from <none> to d871186c2b234914ab63dc5e136f0f03 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:26.670393 17955 raft_consensus.cc:482] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:26.670617 17955 raft_consensus.cc:504] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.671411 17955 leader_election.cc:288] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:26.671844 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 13:59:26.672001 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:26.672075 16974 raft_consensus.cc:2454] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:26.672233 16844 raft_consensus.cc:2454] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:26.672472 16650 leader_election.cc:302] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:26.672719 17955 raft_consensus.cc:2781] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:26.672814 17955 raft_consensus.cc:482] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:26.672906 17955 raft_consensus.cc:3037] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.674805 17955 raft_consensus.cc:504] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.675351 17955 leader_election.cc:288] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Requested vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:26.675917 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:26.675920 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:26.676154 16974 raft_consensus.cc:3037] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.676152 16844 raft_consensus.cc:3037] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.679116 16974 raft_consensus.cc:2454] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:26.679116 16844 raft_consensus.cc:2454] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:26.679562 16650 leader_election.cc:302] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:26.679873 17955 raft_consensus.cc:2781] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:26.680006 17955 raft_consensus.cc:686] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 LEADER]: Becoming Leader. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Running, Role: LEADER
I20240501 13:59:26.680291 17955 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:26.683557 16573 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f8b1516d5142709756d06b06c5d55f (127.16.36.129). New cstate: current_term: 1 leader_uuid: "10f8b1516d5142709756d06b06c5d55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:26.839949 17939 raft_consensus.cc:482] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:26.840169 17939 raft_consensus.cc:504] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.840958 17939 leader_election.cc:288] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:26.841548 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2269a465c771402d882a363a579ba508" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:26.841519 16713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2269a465c771402d882a363a579ba508" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f" is_pre_election: true
I20240501 13:59:26.841828 17104 raft_consensus.cc:2454] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:26.841835 16713 raft_consensus.cc:2454] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:26.842250 16908 leader_election.cc:302] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:26.842540 17939 raft_consensus.cc:2781] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:26.842670 17939 raft_consensus.cc:482] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:26.842775 17939 raft_consensus.cc:3037] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.844594 17939 raft_consensus.cc:504] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.845088 17939 leader_election.cc:288] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Requested vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:26.845521 16713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2269a465c771402d882a363a579ba508" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f"
I20240501 13:59:26.845633 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2269a465c771402d882a363a579ba508" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:26.845711 16713 raft_consensus.cc:3037] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.845870 17104 raft_consensus.cc:3037] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:26.847575 16713 raft_consensus.cc:2454] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:26.847848 17104 raft_consensus.cc:2454] T 2269a465c771402d882a363a579ba508 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:26.847977 16908 leader_election.cc:302] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:26.848270 17939 raft_consensus.cc:2781] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:26.848421 17939 raft_consensus.cc:686] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Becoming Leader. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:26.848699 17939 consensus_queue.cc:237] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:26.851307 16573 catalog_manager.cc:5484] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4b54fe4701746c0930c9b0667c90519 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:26.990742 17726 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:27.001868 16974 raft_consensus.cc:3037] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.007925 17726 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:27.058326 17955 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:27.065840 17955 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:27.232723 17939 consensus_queue.cc:1035] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:27.239881 17999 consensus_queue.cc:1035] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:27.281497 17988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:27.281988 17988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:27.290763 17988 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 290.505 ms
time per row: 0.145252 ms
I20240501 13:59:27.731251 16573 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51592:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:27.740689 16825 tablet_service.cc:1467] Processing CreateTablet for tablet dfb457e2b1b14e9688fd392837c0dc92 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.740885 16953 tablet_service.cc:1467] Processing CreateTablet for tablet dfb457e2b1b14e9688fd392837c0dc92 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.741225 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb457e2b1b14e9688fd392837c0dc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.741263 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb457e2b1b14e9688fd392837c0dc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.741555 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 55f551f8187a49a4afa2e64836a12451 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.741665 16693 tablet_service.cc:1467] Processing CreateTablet for tablet dfb457e2b1b14e9688fd392837c0dc92 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.742017 16693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb457e2b1b14e9688fd392837c0dc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.742208 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f551f8187a49a4afa2e64836a12451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.743486 17085 tablet_service.cc:1467] Processing CreateTablet for tablet 55f551f8187a49a4afa2e64836a12451 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.743923 17085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f551f8187a49a4afa2e64836a12451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.751266 17083 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6e800e4564a63b6a896b63519dca7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.751886 17083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6e800e4564a63b6a896b63519dca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.755856 16954 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6e800e4564a63b6a896b63519dca7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.756450 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6e800e4564a63b6a896b63519dca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.759745 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 55f551f8187a49a4afa2e64836a12451 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.760360 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f551f8187a49a4afa2e64836a12451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.763216 16824 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6e800e4564a63b6a896b63519dca7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:27.763736 16824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6e800e4564a63b6a896b63519dca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:27.767146 18055 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:27.771909 18055 tablet_bootstrap.cc:654] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.778443 18055 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:27.779182 18055 ts_tablet_manager.cc:1367] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 13:59:27.781121 18055 raft_consensus.cc:348] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:27.781898 18055 raft_consensus.cc:374] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.782141 18055 raft_consensus.cc:729] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.782567 18055 consensus_queue.cc:260] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:27.783895 18055 ts_tablet_manager.cc:1398] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:27.784803 18055 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 13:59:27.788944 18055 tablet_bootstrap.cc:654] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.797722 18055 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: No bootstrap required, opened a new log
I20240501 13:59:27.798450 18055 ts_tablet_manager.cc:1367] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240501 13:59:27.799497 18055 raft_consensus.cc:348] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.800915 18055 raft_consensus.cc:374] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.801183 18055 raft_consensus.cc:729] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.801613 18055 consensus_queue.cc:260] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.802809 18055 ts_tablet_manager.cc:1398] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 13:59:27.808624 18053 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:27.810431 18053 tablet_bootstrap.cc:654] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.816044 18053 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:27.816288 18053 ts_tablet_manager.cc:1367] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 13:59:27.817862 18053 raft_consensus.cc:348] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:27.818718 18053 raft_consensus.cc:374] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.818928 18053 raft_consensus.cc:729] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.819334 18053 consensus_queue.cc:260] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:27.820396 18053 ts_tablet_manager.cc:1398] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:27.821262 18053 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:27.823266 18059 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:27.823904 18053 tablet_bootstrap.cc:654] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.825503 18054 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:27.825865 18059 tablet_bootstrap.cc:654] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.827538 18054 tablet_bootstrap.cc:654] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.829286 18053 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:27.829453 18053 ts_tablet_manager.cc:1367] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 13:59:27.830287 18053 raft_consensus.cc:348] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.830526 18053 raft_consensus.cc:374] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.830601 18053 raft_consensus.cc:729] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.830922 18053 consensus_queue.cc:260] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.832213 18053 ts_tablet_manager.cc:1398] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:27.832530 18053 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03: Bootstrap starting.
I20240501 13:59:27.835112 18053 tablet_bootstrap.cc:654] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.855015 18054 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:27.855281 18054 ts_tablet_manager.cc:1367] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.015s
I20240501 13:59:27.856500 18054 raft_consensus.cc:348] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.856830 18054 raft_consensus.cc:374] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.856966 18054 raft_consensus.cc:729] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.860327 18059 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:27.860426 18053 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03: No bootstrap required, opened a new log
I20240501 13:59:27.860601 18053 ts_tablet_manager.cc:1367] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.007s
I20240501 13:59:27.860704 18059 ts_tablet_manager.cc:1367] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.018s
I20240501 13:59:27.861436 18053 raft_consensus.cc:348] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.861661 18053 raft_consensus.cc:374] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.861781 18053 raft_consensus.cc:729] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.861766 18059 raft_consensus.cc:348] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.862112 18059 raft_consensus.cc:374] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.862066 18053 consensus_queue.cc:260] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.862249 18059 raft_consensus.cc:729] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.862676 18059 consensus_queue.cc:260] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.864243 18059 ts_tablet_manager.cc:1398] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:27.864704 18059 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519: Bootstrap starting.
I20240501 13:59:27.863533 18053 ts_tablet_manager.cc:1398] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:27.863782 18054 consensus_queue.cc:260] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.868098 18059 tablet_bootstrap.cc:654] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.869180 18054 ts_tablet_manager.cc:1398] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240501 13:59:27.869532 18054 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: Bootstrap starting.
I20240501 13:59:27.872248 18054 tablet_bootstrap.cc:654] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:27.877204 18059 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519: No bootstrap required, opened a new log
I20240501 13:59:27.877422 18059 ts_tablet_manager.cc:1367] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 13:59:27.878335 18059 raft_consensus.cc:348] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:27.878616 18059 raft_consensus.cc:374] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.878731 18059 raft_consensus.cc:729] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.879030 18059 consensus_queue.cc:260] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:27.879773 18059 ts_tablet_manager.cc:1398] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:27.881232 18054 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: No bootstrap required, opened a new log
I20240501 13:59:27.881395 18054 ts_tablet_manager.cc:1367] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 13:59:27.882486 18054 raft_consensus.cc:348] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.882769 18054 raft_consensus.cc:374] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:27.882870 18054 raft_consensus.cc:729] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Initialized, Role: FOLLOWER
I20240501 13:59:27.884279 17865 raft_consensus.cc:482] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:27.884483 17865 raft_consensus.cc:504] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.884080 18054 consensus_queue.cc:260] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.885180 18054 ts_tablet_manager.cc:1398] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:27.885293 17865 leader_election.cc:288] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:27.885740 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f551f8187a49a4afa2e64836a12451" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:27.885947 17104 raft_consensus.cc:2454] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:27.886059 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f551f8187a49a4afa2e64836a12451" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:27.886319 16844 raft_consensus.cc:2454] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 0.
I20240501 13:59:27.886467 16647 leader_election.cc:302] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, 70ff9aaa9fa2414ea55225aafbd8fd89; no voters:
I20240501 13:59:27.886866 17865 raft_consensus.cc:2781] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:27.886993 17865 raft_consensus.cc:482] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:27.887133 17865 raft_consensus.cc:3037] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.889453 17865 raft_consensus.cc:504] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.890019 17865 leader_election.cc:288] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Requested vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:27.890445 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f551f8187a49a4afa2e64836a12451" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:27.890489 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f551f8187a49a4afa2e64836a12451" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:27.890661 17104 raft_consensus.cc:3037] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.890745 16844 raft_consensus.cc:3037] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.892714 17104 raft_consensus.cc:2454] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:27.892812 16844 raft_consensus.cc:2454] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f8b1516d5142709756d06b06c5d55f in term 1.
I20240501 13:59:27.893141 16647 leader_election.cc:302] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, 70ff9aaa9fa2414ea55225aafbd8fd89; no voters:
I20240501 13:59:27.894901 17865 raft_consensus.cc:2781] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:27.895128 17865 raft_consensus.cc:686] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 1 LEADER]: Becoming Leader. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Running, Role: LEADER
I20240501 13:59:27.895455 17865 consensus_queue.cc:237] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:27.899183 16567 catalog_manager.cc:5484] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f8b1516d5142709756d06b06c5d55f (127.16.36.129). New cstate: current_term: 1 leader_uuid: "10f8b1516d5142709756d06b06c5d55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:27.912032 17999 raft_consensus.cc:482] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:27.912249 17999 raft_consensus.cc:504] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.913012 17999 leader_election.cc:288] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:27.913455 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:27.913573 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:27.913748 17105 raft_consensus.cc:2454] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:27.913794 16844 raft_consensus.cc:2454] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 0.
I20240501 13:59:27.914328 16911 leader_election.cc:302] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:27.914808 17999 raft_consensus.cc:2781] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:27.914893 17999 raft_consensus.cc:482] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:27.914997 17999 raft_consensus.cc:3037] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.916967 17999 raft_consensus.cc:504] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.917655 17999 leader_election.cc:288] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Requested vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:27.918167 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:27.918258 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:27.918440 17105 raft_consensus.cc:3037] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.918485 16844 raft_consensus.cc:3037] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:27.921407 17105 raft_consensus.cc:2454] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:27.921415 16844 raft_consensus.cc:2454] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:27.921846 16911 leader_election.cc:302] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:27.922209 17999 raft_consensus.cc:2781] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:27.922380 17999 raft_consensus.cc:686] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Becoming Leader. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:27.922657 17999 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:27.925724 16567 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4b54fe4701746c0930c9b0667c90519 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:28.048300 17726 raft_consensus.cc:482] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:28.048504 17726 raft_consensus.cc:504] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:28.049155 17726 leader_election.cc:288] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:28.049767 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb457e2b1b14e9688fd392837c0dc92" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 13:59:28.049782 16713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb457e2b1b14e9688fd392837c0dc92" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f" is_pre_election: true
I20240501 13:59:28.050038 16713 raft_consensus.cc:2454] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 0.
I20240501 13:59:28.050033 16975 raft_consensus.cc:2454] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 0.
I20240501 13:59:28.050534 16781 leader_election.cc:302] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:28.050865 17726 raft_consensus.cc:2781] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:28.050956 17726 raft_consensus.cc:482] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:28.051034 17726 raft_consensus.cc:3037] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.053041 17726 raft_consensus.cc:504] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:28.053781 17726 leader_election.cc:288] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 election: Requested vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:28.054296 16713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb457e2b1b14e9688fd392837c0dc92" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f8b1516d5142709756d06b06c5d55f"
I20240501 13:59:28.054366 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb457e2b1b14e9688fd392837c0dc92" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:28.054513 16713 raft_consensus.cc:3037] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.054605 16975 raft_consensus.cc:3037] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.056650 16713 raft_consensus.cc:2454] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:28.056658 16975 raft_consensus.cc:2454] T dfb457e2b1b14e9688fd392837c0dc92 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:28.057116 16778 leader_election.cc:302] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:28.057416 17726 raft_consensus.cc:2781] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:28.057749 17726 raft_consensus.cc:686] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Becoming Leader. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:28.058012 17726 consensus_queue.cc:237] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 13:59:28.060957 16567 catalog_manager.cc:5484] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: term changed from 0 to 1, leader changed from <none> to d871186c2b234914ab63dc5e136f0f03 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:28.381323 17865 consensus_queue.cc:1035] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:28.398648 17938 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:28.398802 18072 consensus_queue.cc:1035] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:28.416285 18080 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:59:28.504531 17726 consensus_queue.cc:1035] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:28.505703 18070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:28.506198 18070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 13:59:28.509649 17726 consensus_queue.cc:1035] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:59:28.520759 18070 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 331.57 ms
time per row: 0.165785 ms
I20240501 13:59:29.011698 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:36049
--local_ip_for_outbound_sockets=127.16.36.133
--tserver_master_addrs=127.16.36.190:43309
--webserver_port=33141
--webserver_interface=127.16.36.133
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 13:59:29.407147 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:29.407503 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:29.407668 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:29.415792 18131 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:29.415920 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:29.416082 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 13:59:29.426318 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:36049
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=33141
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:29.427882 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:29.430068 18131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:29.443775 18137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.444382 18138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.445137 18140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.445624 18131 server_base.cc:1034] running on GCE node
I20240501 13:59:29.446542 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:29.447976 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:29.449249 18131 hybrid_clock.cc:648] HybridClock initialized: now 1714571969449198 us; error 37 us; skew 500 ppm
I20240501 13:59:29.449725 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.453347 18131 webserver.cc:460] Webserver started at http://127.16.36.133:33141/ using document root <none> and password file <none>
I20240501 13:59:29.454509 18131 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.454713 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.463563 18131 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20240501 13:59:29.467929 18146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.469867 18131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240501 13:59:29.470086 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "943a7d1d43144d56b35d62ebf72d1e25"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:29.471537 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.509621 18131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.511005 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:29.511523 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.513065 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:29.515671 18131 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:29.515794 18131 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.515964 18131 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:29.516081 18131 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240501 13:59:29.555223 18131 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:36049
I20240501 13:59:29.555262 18258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:36049 every 8 connection(s)
I20240501 13:59:29.556793 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:29.561512 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 18131
I20240501 13:59:29.562377 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.134:40959
--local_ip_for_outbound_sockets=127.16.36.134
--tserver_master_addrs=127.16.36.190:43309
--webserver_port=40787
--webserver_interface=127.16.36.134
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:29.573104 18259 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:29.573453 18259 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:29.574345 18259 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:29.576313 16570 ts_manager.cc:194] Re-registered known tserver with Master: 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133:36049)
I20240501 13:59:29.577708 16570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:57419
W20240501 13:59:29.931818 18263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:29.932122 18263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:29.932281 18263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:29.939944 18263 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:29.940083 18263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:29.940260 18263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.134
I20240501 13:59:29.949713 18263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.134:40959
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.134
--webserver_port=40787
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.134
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:29.951164 18263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:29.953409 18263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:29.965600 18268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.966202 18269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.966742 18271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.967094 18263 server_base.cc:1034] running on GCE node
I20240501 13:59:29.967731 18263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:29.968737 18263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:29.969965 18263 hybrid_clock.cc:648] HybridClock initialized: now 1714571969969871 us; error 74 us; skew 500 ppm
I20240501 13:59:29.970322 18263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.972892 18263 webserver.cc:460] Webserver started at http://127.16.36.134:40787/ using document root <none> and password file <none>
I20240501 13:59:29.973706 18263 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.973839 18263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.979485 18263 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:29.982470 18277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.983999 18263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 13:59:29.984154 18263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "200e98827f7c44d29cd71ffd7faec1ca"
format_stamp: "Formatted at 2024-05-01 13:59:16 on dist-test-slave-9q9t"
I20240501 13:59:29.985069 18263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:30.008615 18263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:30.009546 18263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:30.009846 18263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:30.010900 18263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:30.012562 18263 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:30.012637 18263 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.012748 18263 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:30.012791 18263 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.046310 18263 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.134:40959
I20240501 13:59:30.046360 18389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.134:40959 every 8 connection(s)
I20240501 13:59:30.048004 18263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:30.054860 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 18263
I20240501 13:59:30.055560 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.135:39399
--local_ip_for_outbound_sockets=127.16.36.135
--tserver_master_addrs=127.16.36.190:43309
--webserver_port=39357
--webserver_interface=127.16.36.135
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:30.061897 18390 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:30.062296 18390 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:30.063444 18390 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:30.065150 16570 ts_manager.cc:194] Re-registered known tserver with Master: 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959)
I20240501 13:59:30.065986 16570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.134:44199
W20240501 13:59:30.431787 18393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:30.432119 18393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:30.432263 18393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:30.440989 18393 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:30.441116 18393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:30.441278 18393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.135
I20240501 13:59:30.451764 18393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.135:39399
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.135
--webserver_port=39357
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.135
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:30.453271 18393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:30.455387 18393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:30.469434 18399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.469774 18398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.470400 18401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:30.471027 18393 server_base.cc:1034] running on GCE node
I20240501 13:59:30.471715 18393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:30.472808 18393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:30.474028 18393 hybrid_clock.cc:648] HybridClock initialized: now 1714571970473966 us; error 54 us; skew 500 ppm
I20240501 13:59:30.474450 18393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:30.477099 18393 webserver.cc:460] Webserver started at http://127.16.36.135:39357/ using document root <none> and password file <none>
I20240501 13:59:30.477922 18393 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:30.478070 18393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:30.484866 18393 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:30.488363 18407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.490051 18393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:30.490226 18393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "efb8dd25e14a46ae89eda6bbe5f940a2"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:30.491313 18393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:30.510146 18393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:30.511098 18393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:30.511423 18393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:30.512398 18393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:30.514129 18393 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:30.514221 18393 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.514325 18393 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:30.514370 18393 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.550338 18393 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.135:39399
I20240501 13:59:30.550382 18519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.135:39399 every 8 connection(s)
I20240501 13:59:30.551990 18393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:30.557513 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 18393
I20240501 13:59:30.558159 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.136:40683
--local_ip_for_outbound_sockets=127.16.36.136
--tserver_master_addrs=127.16.36.190:43309
--webserver_port=43915
--webserver_interface=127.16.36.136
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 13:59:30.568560 18520 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:30.568965 18520 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:30.570210 18520 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:30.572293 16570 ts_manager.cc:194] Re-registered known tserver with Master: efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399)
I20240501 13:59:30.573220 16570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.135:39985
I20240501 13:59:30.580858 18259 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
W20240501 13:59:30.932533 18523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:30.932870 18523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:30.933060 18523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:30.940959 18523 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:30.941102 18523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:30.941287 18523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.136
I20240501 13:59:30.952783 18523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.136:40683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.136
--webserver_port=43915
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.136
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:30.954277 18523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:30.956239 18523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:30.969624 18529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.970538 18530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.970949 18532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:30.971274 18523 server_base.cc:1034] running on GCE node
I20240501 13:59:30.971944 18523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:30.972946 18523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:30.974170 18523 hybrid_clock.cc:648] HybridClock initialized: now 1714571970974116 us; error 49 us; skew 500 ppm
I20240501 13:59:30.974529 18523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:30.977043 18523 webserver.cc:460] Webserver started at http://127.16.36.136:43915/ using document root <none> and password file <none>
I20240501 13:59:30.977841 18523 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:30.977988 18523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:30.984284 18523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 13:59:30.987874 18538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.989369 18523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:30.989547 18523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2e5565b8761548539de35f9318bec337"
format_stamp: "Formatted at 2024-05-01 13:59:17 on dist-test-slave-9q9t"
I20240501 13:59:30.990474 18523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:31.012440 18523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:31.013442 18523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:31.013757 18523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:31.014931 18523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:31.016746 18523 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:31.016831 18523 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:31.016975 18523 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:31.017033 18523 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:31.058141 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.136:40683 every 8 connection(s)
I20240501 13:59:31.058140 18523 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.136:40683
I20240501 13:59:31.059911 18523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 13:59:31.060616 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 18523
I20240501 13:59:31.069362 18390 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
I20240501 13:59:31.076582 18651 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 13:59:31.077153 18651 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:31.078126 18651 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
I20240501 13:59:31.079761 16570 ts_manager.cc:194] Re-registered known tserver with Master: 2e5565b8761548539de35f9318bec337 (127.16.36.136:40683)
I20240501 13:59:31.080651 16570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.136:38587
I20240501 13:59:31.577226 18520 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
I20240501 13:59:31.636382 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:31.645015 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:31.649122 16693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:31.666589 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:31.681144 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:31.735419 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:31.790509 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:31.812765 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:32.083325 18651 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
I20240501 13:59:32.400315 16573 ts_manager.cc:292] Set tserver state for 10f8b1516d5142709756d06b06c5d55f to MAINTENANCE_MODE
W20240501 13:59:33.855222 18722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:33.855587 18722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:33.865607 18722 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 13:59:34.111065 16693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:34.121166 16693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 13:59:34.124217 16911 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:42745, user_credentials={real_user=slave}} blocked reactor thread for 53598.2us
I20240501 13:59:34.174903 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.186899 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.212877 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.217962 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:34.218575 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.223907 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:34.255298 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:34.267365 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.304106 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:34.308082 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.337397 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:34.351871 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:34.357894 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.368669 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.847687 16693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:34.881582 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.887887 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:34.938120 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:34.947054 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:34.966599 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.015975 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.017293 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 13:59:35.251538 18829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:35.252014 18829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 13:59:35.253319 16693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 13:59:35.264367 18829 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 13:59:35.281706 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:35.297142 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.306690 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:35.333185 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:35.352279 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.352486 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.362413 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.476032 16714 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:35.479290 16844 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer 10f8b1516d5142709756d06b06c5d55f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:35.479425 16975 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Refusing update from remote peer 10f8b1516d5142709756d06b06c5d55f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:35.480120 18852 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:35.480597 18916 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:35.485190 18852 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } }
I20240501 13:59:35.489687 16844 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } }
I20240501 13:59:35.496878 16975 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } }
I20240501 13:59:35.502147 16572 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "10f8b1516d5142709756d06b06c5d55f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:35.521586 16714 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:35.526381 16975 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Refusing update from remote peer 10f8b1516d5142709756d06b06c5d55f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:35.527194 16844 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer 10f8b1516d5142709756d06b06c5d55f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:35.527420 18852 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:35.531672 18852 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:35.538295 18852 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:35.542322 16975 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:35.543598 16844 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:35.549610 16565 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 4 to 5, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "10f8b1516d5142709756d06b06c5d55f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:35.553493 16557 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 101e3579d2954f2aac17d0f10ac84865 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 13:59:35.562304 16647 consensus_peers.cc:485] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f -> Peer 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133:36049): Couldn't send request to peer 943a7d1d43144d56b35d62ebf72d1e25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101e3579d2954f2aac17d0f10ac84865. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:35.638368 16693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:35.662977 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.669947 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:35.675952 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.683240 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.698776 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:35.700258 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:35.739809 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:36.086243 18968 ts_tablet_manager.cc:902] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: Initiating tablet copy from peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:36.090181 18968 tablet_copy_client.cc:308] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:35177
I20240501 13:59:36.119271 16734 tablet_copy_service.cc:140] P 10f8b1516d5142709756d06b06c5d55f: Received BeginTabletCopySession request for tablet 101e3579d2954f2aac17d0f10ac84865 from peer 943a7d1d43144d56b35d62ebf72d1e25 ({username='slave'} at 127.16.36.133:37551)
I20240501 13:59:36.119588 16734 tablet_copy_service.cc:161] P 10f8b1516d5142709756d06b06c5d55f: Beginning new tablet copy session on tablet 101e3579d2954f2aac17d0f10ac84865 from peer 943a7d1d43144d56b35d62ebf72d1e25 at {username='slave'} at 127.16.36.133:37551: session id = 943a7d1d43144d56b35d62ebf72d1e25-101e3579d2954f2aac17d0f10ac84865
I20240501 13:59:36.123917 16734 tablet_copy_source_session.cc:205] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:36.129459 18968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101e3579d2954f2aac17d0f10ac84865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.141136 18968 tablet_copy_client.cc:783] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:36.141573 18968 tablet_copy_client.cc:647] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:36.144559 18972 tablet_copy_client.cc:1002] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Injecting latency into file download: 1500
W20240501 13:59:36.330192 18962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:36.330559 18962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:36.338503 18962 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 13:59:36.512925 16693 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:36.513695 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.516292 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.531754 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.566753 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:36.584147 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:36.585178 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:36.674320 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240501 13:59:36.873171 18720 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 16632
I20240501 13:59:36.882064 16754 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:37.060714 18720 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 16632
W20240501 13:59:37.092993 16911 connection.cc:535] server connection from 127.16.36.129:51833 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 13:59:37.093189 16779 connection.cc:535] server connection from 127.16.36.129:51425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 13:59:37.093989 16778 connection.cc:535] client connection to 127.16.36.129:35177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 13:59:37.094336 16778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 13:59:37.097879 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:37.098170 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:37.123831 16908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
W20240501 13:59:37.125497 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:37.157892 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:37.209729 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 13:59:37.212956 17038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:37.228250 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:37.228605 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 13:59:37.228689 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:37.233892 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:37.244014 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:37.245795 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:37.295701 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:37.473459 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:37.475282 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:37.496704 16824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:37.501838 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:37.515357 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:37.515990 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:37.531522 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 13:59:37.578462 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:37.631858 16844 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:37.635058 16975 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:37.635839 17105 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:37.635890 19091 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:37.637084 17726 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:37.640255 19091 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:37.641010 16975 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:37.641955 17105 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:37.645154 18968 tablet_copy_client.cc:523] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 13:59:37.645846 16780 consensus_peers.cc:485] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 -> Peer 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959): Couldn't send request to peer 200e98827f7c44d29cd71ffd7faec1ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f408fdebfb84082b0a55ad059e90694. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:37.648856 16570 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: config changed from index -1 to 4, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:37.650157 18968 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap starting.
I20240501 13:59:37.663705 18968 log.cc:824] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:37.679957 16975 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:37.683764 17105 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:37.685644 19017 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 13:59:37.692955 16908 consensus_peers.cc:485] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 -> Peer efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399): Couldn't send request to peer efb8dd25e14a46ae89eda6bbe5f940a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d6e800e4564a63b6a896b63519dca7. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:37.695307 16844 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:37.696002 19149 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:37.697207 19151 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:37.698069 17104 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:37.705011 16573 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:37.711014 16844 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:37.727097 18968 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:37.727666 18968 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap complete.
I20240501 13:59:37.728132 18968 ts_tablet_manager.cc:1367] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: Time spent bootstrapping tablet: real 0.078s user 0.063s sys 0.010s
I20240501 13:59:37.734102 18968 raft_consensus.cc:348] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:37.734678 18968 raft_consensus.cc:729] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 943a7d1d43144d56b35d62ebf72d1e25, State: Initialized, Role: LEARNER
I20240501 13:59:37.735244 18968 consensus_queue.cc:260] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:37.737421 18968 ts_tablet_manager.cc:1398] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
W20240501 13:59:37.739542 18968 tablet_copy_client.cc:1108] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:37.984884 19166 raft_consensus.cc:482] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:37.985157 19166 raft_consensus.cc:504] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:37.985961 19166 leader_election.cc:288] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:37.986599 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f551f8187a49a4afa2e64836a12451" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
W20240501 13:59:37.987565 17038 leader_election.cc:334] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:37.987702 17038 leader_election.cc:302] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89; no voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03
I20240501 13:59:37.987993 19166 raft_consensus.cc:2726] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:59:38.064649 19166 raft_consensus.cc:482] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.064890 19166 raft_consensus.cc:504] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:38.065619 19166 leader_election.cc:288] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:38.066330 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27c70cad684568af8d8119559475cc" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
W20240501 13:59:38.067234 17038 leader_election.cc:334] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.067350 17038 leader_election.cc:302] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89; no voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03
I20240501 13:59:38.067622 19166 raft_consensus.cc:2726] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:59:38.082533 19139 raft_consensus.cc:482] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.082820 19139 raft_consensus.cc:504] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:38.083571 19139 leader_election.cc:288] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.084231 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3068f422a4d54d998715bffa2667854e" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
W20240501 13:59:38.085494 16778 leader_election.cc:334] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.085645 16778 leader_election.cc:302] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03; no voters: 10f8b1516d5142709756d06b06c5d55f, f4b54fe4701746c0930c9b0667c90519
I20240501 13:59:38.085947 19139 raft_consensus.cc:2726] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:59:38.181242 19189 ts_tablet_manager.cc:902] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: Initiating tablet copy from peer d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:38.183612 19189 tablet_copy_client.cc:308] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:42745
I20240501 13:59:38.185878 19151 raft_consensus.cc:482] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.186110 19151 raft_consensus.cc:504] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:38.186875 19151 leader_election.cc:288] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:38.187759 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
W20240501 13:59:38.188174 16908 leader_election.cc:334] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.188589 16911 leader_election.cc:302] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4b54fe4701746c0930c9b0667c90519; no voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03
I20240501 13:59:38.189031 19151 raft_consensus.cc:2726] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:59:38.195578 16865 tablet_copy_service.cc:140] P d871186c2b234914ab63dc5e136f0f03: Received BeginTabletCopySession request for tablet 5f408fdebfb84082b0a55ad059e90694 from peer 200e98827f7c44d29cd71ffd7faec1ca ({username='slave'} at 127.16.36.134:60115)
I20240501 13:59:38.195868 16865 tablet_copy_service.cc:161] P d871186c2b234914ab63dc5e136f0f03: Beginning new tablet copy session on tablet 5f408fdebfb84082b0a55ad059e90694 from peer 200e98827f7c44d29cd71ffd7faec1ca at {username='slave'} at 127.16.36.134:60115: session id = 200e98827f7c44d29cd71ffd7faec1ca-5f408fdebfb84082b0a55ad059e90694
I20240501 13:59:38.198920 16865 tablet_copy_source_session.cc:205] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:38.203922 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f408fdebfb84082b0a55ad059e90694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.216260 19189 tablet_copy_client.cc:783] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:38.216650 19189 tablet_copy_client.cc:647] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:38.219099 19193 tablet_copy_client.cc:1002] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:38.224669 19196 ts_tablet_manager.cc:902] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: Initiating tablet copy from peer f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:38.226612 19196 tablet_copy_client.cc:308] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:37163
I20240501 13:59:38.232007 19139 raft_consensus.cc:482] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.232183 19139 raft_consensus.cc:504] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:38.232903 19139 leader_election.cc:288] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:38.233733 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27c70cad684568af8d8119559475cc" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:38.233969 17104 raft_consensus.cc:2454] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:38.234440 16778 leader_election.cc:302] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:38.234890 19139 raft_consensus.cc:2781] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:59:38.234802 16995 tablet_copy_service.cc:140] P f4b54fe4701746c0930c9b0667c90519: Received BeginTabletCopySession request for tablet d4d6e800e4564a63b6a896b63519dca7 from peer efb8dd25e14a46ae89eda6bbe5f940a2 ({username='slave'} at 127.16.36.135:60491)
I20240501 13:59:38.235039 19139 raft_consensus.cc:482] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.235095 16995 tablet_copy_service.cc:161] P f4b54fe4701746c0930c9b0667c90519: Beginning new tablet copy session on tablet d4d6e800e4564a63b6a896b63519dca7 from peer efb8dd25e14a46ae89eda6bbe5f940a2 at {username='slave'} at 127.16.36.135:60491: session id = efb8dd25e14a46ae89eda6bbe5f940a2-d4d6e800e4564a63b6a896b63519dca7
I20240501 13:59:38.235153 19139 raft_consensus.cc:3037] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:38.235906 16778 leader_election.cc:334] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.237308 19139 raft_consensus.cc:504] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:38.237838 19139 leader_election.cc:288] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Requested vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:38.238396 16995 tablet_copy_source_session.cc:205] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:38.238858 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f27c70cad684568af8d8119559475cc" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:38.239099 17104 raft_consensus.cc:3037] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:38.239320 16778 leader_election.cc:334] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.241680 17104 raft_consensus.cc:2454] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 2.
I20240501 13:59:38.242236 16778 leader_election.cc:302] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters: 10f8b1516d5142709756d06b06c5d55f
I20240501 13:59:38.242576 19139 raft_consensus.cc:2781] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:38.243014 19139 raft_consensus.cc:686] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Becoming Leader. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:38.243386 19139 consensus_queue.cc:237] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 13:59:38.244578 19196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6e800e4564a63b6a896b63519dca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.246414 16570 catalog_manager.cc:5484] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 reported cstate change: term changed from 1 to 2, leader changed from 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) to d871186c2b234914ab63dc5e136f0f03 (127.16.36.130). New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:38.256345 19196 tablet_copy_client.cc:783] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:38.256673 19196 tablet_copy_client.cc:647] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:38.259131 19200 tablet_copy_client.cc:1002] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:38.303480 19166 raft_consensus.cc:482] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.303709 19166 raft_consensus.cc:504] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:38.304459 19166 leader_election.cc:288] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.305032 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c8c82da2da4173899f5b9e03894da8" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
W20240501 13:59:38.306105 17038 leader_election.cc:334] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.306260 17038 leader_election.cc:302] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89; no voters: 10f8b1516d5142709756d06b06c5d55f, d871186c2b234914ab63dc5e136f0f03
I20240501 13:59:38.306548 19166 raft_consensus.cc:2726] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:59:38.333447 19139 raft_consensus.cc:482] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.333686 19139 raft_consensus.cc:504] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:38.334323 19139 leader_election.cc:288] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.335001 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f551f8187a49a4afa2e64836a12451" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:38.335255 17104 raft_consensus.cc:2454] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:38.335786 16778 leader_election.cc:302] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters:
W20240501 13:59:38.336091 16778 leader_election.cc:334] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.336133 19139 raft_consensus.cc:2781] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:59:38.336243 19139 raft_consensus.cc:482] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.336359 19139 raft_consensus.cc:3037] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:38.338224 19139 raft_consensus.cc:504] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:38.338876 19139 leader_election.cc:288] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Requested vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.339283 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f551f8187a49a4afa2e64836a12451" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:38.339521 17104 raft_consensus.cc:3037] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:38.340101 16778 leader_election.cc:334] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.341500 17104 raft_consensus.cc:2454] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 2.
I20240501 13:59:38.341720 19139 raft_consensus.cc:482] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.341897 19139 raft_consensus.cc:504] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:38.341975 16778 leader_election.cc:302] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters: 10f8b1516d5142709756d06b06c5d55f
I20240501 13:59:38.342293 19202 raft_consensus.cc:2781] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:38.342702 19139 leader_election.cc:288] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.342815 19202 raft_consensus.cc:686] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Becoming Leader. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:38.343116 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 13:59:38.343350 16975 raft_consensus.cc:2454] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:38.343178 19202 consensus_queue.cc:237] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:38.343847 16781 leader_election.cc:302] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:38.344162 19139 raft_consensus.cc:2781] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:59:38.344255 19139 raft_consensus.cc:482] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.344383 19139 raft_consensus.cc:3037] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:38.344679 16778 leader_election.cc:334] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.346949 19139 raft_consensus.cc:504] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:38.347290 16570 catalog_manager.cc:5484] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: term changed from 1 to 2, leader changed from 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) to d871186c2b234914ab63dc5e136f0f03 (127.16.36.130). New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:38.347615 19139 leader_election.cc:288] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Requested vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.348017 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:38.348269 16975 raft_consensus.cc:3037] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:38.349393 16778 leader_election.cc:334] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.350886 16975 raft_consensus.cc:2454] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 2.
I20240501 13:59:38.351359 16781 leader_election.cc:302] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters: 10f8b1516d5142709756d06b06c5d55f
I20240501 13:59:38.351650 19139 raft_consensus.cc:2781] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:38.351819 19139 raft_consensus.cc:686] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Becoming Leader. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:38.352121 19139 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:38.355176 16570 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: term changed from 1 to 2, leader changed from 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) to d871186c2b234914ab63dc5e136f0f03 (127.16.36.130). New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:38.561750 19151 raft_consensus.cc:482] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.562080 19139 raft_consensus.cc:482] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.562014 19151 raft_consensus.cc:504] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:38.562259 19139 raft_consensus.cc:504] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:38.562924 19151 leader_election.cc:288] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.563040 19139 leader_election.cc:288] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:38.563477 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3068f422a4d54d998715bffa2667854e" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 13:59:38.563710 16844 raft_consensus.cc:2454] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 1.
I20240501 13:59:38.564062 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c8c82da2da4173899f5b9e03894da8" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" is_pre_election: true
I20240501 13:59:38.564226 16911 leader_election.cc:302] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:38.564311 17104 raft_consensus.cc:2454] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 1.
I20240501 13:59:38.564564 19151 raft_consensus.cc:2781] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:59:38.564675 19151 raft_consensus.cc:482] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.564819 19151 raft_consensus.cc:3037] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:38.564934 16778 leader_election.cc:334] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
W20240501 13:59:38.564950 16908 leader_election.cc:334] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.565227 16778 leader_election.cc:302] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters: 10f8b1516d5142709756d06b06c5d55f
I20240501 13:59:38.565515 19139 raft_consensus.cc:2781] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:59:38.565636 19139 raft_consensus.cc:482] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10f8b1516d5142709756d06b06c5d55f)
I20240501 13:59:38.565730 19139 raft_consensus.cc:3037] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:38.567888 19151 raft_consensus.cc:504] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:38.568529 19139 raft_consensus.cc:504] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:38.568730 19151 leader_election.cc:288] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 election: Requested vote from peers d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 13:59:38.569258 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3068f422a4d54d998715bffa2667854e" candidate_uuid: "f4b54fe4701746c0930c9b0667c90519" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:38.569471 16844 raft_consensus.cc:3037] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:38.569890 19139 leader_election.cc:288] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Requested vote from peers 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177), 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:38.570565 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c8c82da2da4173899f5b9e03894da8" candidate_uuid: "d871186c2b234914ab63dc5e136f0f03" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:38.570832 17104 raft_consensus.cc:3037] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:38.570878 16778 leader_election.cc:334] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
W20240501 13:59:38.570948 16908 leader_election.cc:334] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:38.572055 16844 raft_consensus.cc:2454] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4b54fe4701746c0930c9b0667c90519 in term 2.
I20240501 13:59:38.572499 16911 leader_election.cc:302] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519; no voters: 10f8b1516d5142709756d06b06c5d55f
I20240501 13:59:38.572810 19151 raft_consensus.cc:2781] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:38.572999 19151 raft_consensus.cc:686] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [term 2 LEADER]: Becoming Leader. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:38.573071 17104 raft_consensus.cc:2454] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d871186c2b234914ab63dc5e136f0f03 in term 2.
I20240501 13:59:38.573330 19151 consensus_queue.cc:237] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 13:59:38.573535 16778 leader_election.cc:302] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters: 10f8b1516d5142709756d06b06c5d55f
I20240501 13:59:38.573851 19139 raft_consensus.cc:2781] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:38.574105 19139 raft_consensus.cc:686] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Becoming Leader. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:38.574411 19139 consensus_queue.cc:237] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 13:59:38.577387 16570 catalog_manager.cc:5484] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 reported cstate change: term changed from 1 to 2, leader changed from 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) to f4b54fe4701746c0930c9b0667c90519 (127.16.36.131). New cstate: current_term: 2 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:38.578188 16573 catalog_manager.cc:5484] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: term changed from 1 to 2, leader changed from 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) to d871186c2b234914ab63dc5e136f0f03 (127.16.36.130). New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:38.722172 17104 raft_consensus.cc:1261] T 8f27c70cad684568af8d8119559475cc P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 13:59:38.722788 19139 consensus_queue.cc:1035] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 13:59:38.728688 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:38.730244 17104 raft_consensus.cc:1261] T 55f551f8187a49a4afa2e64836a12451 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 13:59:38.730794 19207 consensus_queue.cc:1035] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 13:59:38.736512 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:38.843493 18214 raft_consensus.cc:3037] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Advancing to term 2
I20240501 13:59:38.846182 18214 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 LEARNER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:38.847196 19139 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:38.851310 19139 raft_consensus.cc:1050] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: attempting to promote NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 to VOTER
W20240501 13:59:38.851598 19139 raft_consensus.cc:1054] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: Unable to promote non-voter 943a7d1d43144d56b35d62ebf72d1e25: Illegal state: Leader has not yet committed an operation in its own term
I20240501 13:59:38.891542 16975 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:38.892164 19202 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 13:59:38.899343 16778 consensus_peers.cc:485] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:38.900633 19207 raft_consensus.cc:1050] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: attempting to promote NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 to VOTER
I20240501 13:59:38.901679 19207 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:38.903776 16975 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 13:59:38.903777 18214 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 LEARNER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 13:59:38.904481 19202 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 13:59:38.904773 19139 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 13:59:38.905081 16778 consensus_peers.cc:485] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:38.907554 19202 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:38.908283 18214 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:38.912371 16975 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:38.913167 16565 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 5 to 7, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:38.918449 16844 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:38.920655 18214 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 13:59:38.920763 16975 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 13:59:38.921382 19139 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:38.922185 19139 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:38.924667 19207 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:38.925391 18214 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:38.925359 16975 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:38.928853 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 101e3579d2954f2aac17d0f10ac84865 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:38.930308 16573 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 reported cstate change: config changed from index 7 to 8, VOTER 10f8b1516d5142709756d06b06c5d55f (127.16.36.129) evicted. New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
W20240501 13:59:38.935628 16557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
W20240501 13:59:38.937234 16557 catalog_manager.cc:4573] TS 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 101e3579d2954f2aac17d0f10ac84865: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
W20240501 13:59:38.954274 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:39.017418 16844 raft_consensus.cc:1261] T 3068f422a4d54d998715bffa2667854e P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 13:59:39.018117 19151 consensus_queue.cc:1035] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:39.068152 17104 raft_consensus.cc:1261] T f9c8c82da2da4173899f5b9e03894da8 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 13:59:39.068946 19207 consensus_queue.cc:1035] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 13:59:39.075343 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:39.169421 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 13:59:39.550455 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 13:59:39.719761 19189 tablet_copy_client.cc:523] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:39.723541 19189 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap starting.
I20240501 13:59:39.735419 19189 log.cc:824] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:39.759711 19196 tablet_copy_client.cc:523] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:39.765353 19196 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap starting.
I20240501 13:59:39.779183 19196 log.cc:824] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:39.780355 19189 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:39.780963 19189 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap complete.
I20240501 13:59:39.781499 19189 ts_tablet_manager.cc:1367] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: Time spent bootstrapping tablet: real 0.058s user 0.054s sys 0.004s
W20240501 13:59:39.782088 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 13:59:39.788148 19189 raft_consensus.cc:348] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:39.788681 19189 raft_consensus.cc:729] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 200e98827f7c44d29cd71ffd7faec1ca, State: Initialized, Role: LEARNER
I20240501 13:59:39.789299 19189 consensus_queue.cc:260] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:39.791252 19189 ts_tablet_manager.cc:1398] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240501 13:59:39.792395 16865 tablet_copy_service.cc:342] P d871186c2b234914ab63dc5e136f0f03: Request end of tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-5f408fdebfb84082b0a55ad059e90694 received from {username='slave'} at 127.16.36.134:60115
I20240501 13:59:39.792591 16865 tablet_copy_service.cc:433] P d871186c2b234914ab63dc5e136f0f03: ending tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-5f408fdebfb84082b0a55ad059e90694 on tablet 5f408fdebfb84082b0a55ad059e90694 with peer 200e98827f7c44d29cd71ffd7faec1ca
W20240501 13:59:39.807188 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 13:59:39.818374 19196 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:39.819041 19196 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap complete.
I20240501 13:59:39.819619 19196 ts_tablet_manager.cc:1367] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent bootstrapping tablet: real 0.055s user 0.050s sys 0.004s
I20240501 13:59:39.826360 19196 raft_consensus.cc:348] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:39.826977 19196 raft_consensus.cc:729] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efb8dd25e14a46ae89eda6bbe5f940a2, State: Initialized, Role: LEARNER
I20240501 13:59:39.827649 19196 consensus_queue.cc:260] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:39.829594 19196 ts_tablet_manager.cc:1398] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240501 13:59:39.830737 16995 tablet_copy_service.cc:342] P f4b54fe4701746c0930c9b0667c90519: Request end of tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-d4d6e800e4564a63b6a896b63519dca7 received from {username='slave'} at 127.16.36.135:60491
I20240501 13:59:39.830948 16995 tablet_copy_service.cc:433] P f4b54fe4701746c0930c9b0667c90519: ending tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-d4d6e800e4564a63b6a896b63519dca7 on tablet d4d6e800e4564a63b6a896b63519dca7 with peer efb8dd25e14a46ae89eda6bbe5f940a2
W20240501 13:59:40.043368 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 13:59:40.074353 18345 raft_consensus.cc:1203] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 13:59:40.140811 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 13:59:40.152029 16557 catalog_manager.cc:4573] TS 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 101e3579d2954f2aac17d0f10ac84865: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:40.279771 18471 raft_consensus.cc:1203] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:40.514412 19342 raft_consensus.cc:1050] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: attempting to promote NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca to VOTER
I20240501 13:59:40.515417 19342 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:40.517956 17105 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:40.518088 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:40.518216 16975 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:40.518829 19341 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:40.519188 19344 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:40.519508 19342 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:40.523432 19341 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.524128 17105 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.525245 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.529004 16975 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.529971 16565 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 4 to 5, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.537827 16844 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:40.539973 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:40.540099 17105 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:40.540586 19342 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:40.541000 19344 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:40.543447 19342 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.544271 17105 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.544559 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.548233 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f408fdebfb84082b0a55ad059e90694 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:40.549053 16569 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:40.553244 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f408fdebfb84082b0a55ad059e90694 with delete_type TABLET_DATA_TOMBSTONED (TS f4b54fe4701746c0930c9b0667c90519 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41412
I20240501 13:59:40.554251 19352 tablet_replica.cc:330] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: stopping tablet replica
I20240501 13:59:40.554796 19352 raft_consensus.cc:2229] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:40.555255 19352 raft_consensus.cc:2258] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:40.557583 19352 ts_tablet_manager.cc:1875] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:40.564882 19352 ts_tablet_manager.cc:1888] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 13:59:40.565158 19352 log.cc:1196] T 5f408fdebfb84082b0a55ad059e90694 P f4b54fe4701746c0930c9b0667c90519: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f408fdebfb84082b0a55ad059e90694
I20240501 13:59:40.566294 16560 catalog_manager.cc:4830] TS f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163): tablet 5f408fdebfb84082b0a55ad059e90694 (table rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]) successfully deleted
I20240501 13:59:40.705215 19370 raft_consensus.cc:1050] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: attempting to promote NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 to VOTER
I20240501 13:59:40.706068 19370 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:40.709028 17105 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:40.709013 16844 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:40.709128 18471 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:40.709690 19373 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:40.710085 19370 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:40.710361 19017 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:40.715694 19017 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.716444 17105 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.716707 18471 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.719717 16844 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.720655 16573 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:40.727062 16975 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:40.728979 18471 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:40.728977 17105 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:40.729558 19371 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:40.729864 19017 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:40.732290 19373 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.733285 18471 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.735594 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4d6e800e4564a63b6a896b63519dca7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:40.735985 17105 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.738371 16570 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 reported cstate change: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:40.743119 16824 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6e800e4564a63b6a896b63519dca7 with delete_type TABLET_DATA_TOMBSTONED (TS d871186c2b234914ab63dc5e136f0f03 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42466
I20240501 13:59:40.744184 19377 tablet_replica.cc:330] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: stopping tablet replica
I20240501 13:59:40.744585 19377 raft_consensus.cc:2229] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:40.744968 19377 raft_consensus.cc:2258] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:40.746704 19377 ts_tablet_manager.cc:1875] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:40.751780 19377 ts_tablet_manager.cc:1888] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 13:59:40.751938 19377 log.cc:1196] T d4d6e800e4564a63b6a896b63519dca7 P d871186c2b234914ab63dc5e136f0f03: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/d4d6e800e4564a63b6a896b63519dca7
I20240501 13:59:40.752768 16560 catalog_manager.cc:4830] TS d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745): tablet d4d6e800e4564a63b6a896b63519dca7 (table rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]) successfully deleted
I20240501 13:59:40.894686 16975 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } }
I20240501 13:59:40.897840 17105 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:40.898569 16844 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:40.899178 19017 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:40.899557 19373 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 13:59:40.909246 16910 consensus_peers.cc:485] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 -> Peer 2e5565b8761548539de35f9318bec337 (127.16.36.136:40683): Couldn't send request to peer 2e5565b8761548539de35f9318bec337. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 221688d4e2374a42b03e96d89166e0ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:40.910645 19017 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } } }
I20240501 13:59:40.911711 16844 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } } }
I20240501 13:59:40.912442 17105 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } } }
I20240501 13:59:40.917053 16569 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } } }
I20240501 13:59:40.935092 17105 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:40.938071 16975 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:40.939800 19166 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:40.941160 16844 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:40.941817 19166 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:40.943550 19399 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:40.944432 16975 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
W20240501 13:59:40.949997 17038 consensus_peers.cc:485] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399): Couldn't send request to peer efb8dd25e14a46ae89eda6bbe5f940a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3bfe952008f438c887a7d9574fc8bb9. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:40.949936 16569 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:40.952757 16845 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
W20240501 13:59:41.020292 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 13:59:41.185020 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 13:59:41.221037 16557 catalog_manager.cc:4573] TS 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 101e3579d2954f2aac17d0f10ac84865: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
W20240501 13:59:41.256675 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 13:59:41.448715 19437 ts_tablet_manager.cc:902] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: Initiating tablet copy from peer 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:41.450284 19437 tablet_copy_client.cc:308] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:37337
I20240501 13:59:41.456743 17125 tablet_copy_service.cc:140] P 70ff9aaa9fa2414ea55225aafbd8fd89: Received BeginTabletCopySession request for tablet a3bfe952008f438c887a7d9574fc8bb9 from peer efb8dd25e14a46ae89eda6bbe5f940a2 ({username='slave'} at 127.16.36.135:39913)
I20240501 13:59:41.456967 17125 tablet_copy_service.cc:161] P 70ff9aaa9fa2414ea55225aafbd8fd89: Beginning new tablet copy session on tablet a3bfe952008f438c887a7d9574fc8bb9 from peer efb8dd25e14a46ae89eda6bbe5f940a2 at {username='slave'} at 127.16.36.135:39913: session id = efb8dd25e14a46ae89eda6bbe5f940a2-a3bfe952008f438c887a7d9574fc8bb9
I20240501 13:59:41.460120 17125 tablet_copy_source_session.cc:205] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:41.461884 19437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfe952008f438c887a7d9574fc8bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.466503 19437 tablet_copy_client.cc:783] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:41.466797 19437 tablet_copy_client.cc:647] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:41.468750 19441 tablet_copy_client.cc:1002] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:41.508706 19443 ts_tablet_manager.cc:902] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: Initiating tablet copy from peer f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:41.510766 19443 tablet_copy_client.cc:308] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:37163
I20240501 13:59:41.518534 16995 tablet_copy_service.cc:140] P f4b54fe4701746c0930c9b0667c90519: Received BeginTabletCopySession request for tablet 221688d4e2374a42b03e96d89166e0ee from peer 2e5565b8761548539de35f9318bec337 ({username='slave'} at 127.16.36.136:44671)
I20240501 13:59:41.518767 16995 tablet_copy_service.cc:161] P f4b54fe4701746c0930c9b0667c90519: Beginning new tablet copy session on tablet 221688d4e2374a42b03e96d89166e0ee from peer 2e5565b8761548539de35f9318bec337 at {username='slave'} at 127.16.36.136:44671: session id = 2e5565b8761548539de35f9318bec337-221688d4e2374a42b03e96d89166e0ee
I20240501 13:59:41.521435 16995 tablet_copy_source_session.cc:205] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:41.526446 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221688d4e2374a42b03e96d89166e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:41.537199 19443 tablet_copy_client.cc:783] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:41.537613 19443 tablet_copy_client.cc:647] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:41.540023 19447 tablet_copy_client.cc:1002] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: tablet copy: Injecting latency into file download: 1500
W20240501 13:59:41.560642 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 13:59:41.673345 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 13:59:42.143421 16778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 13:59:42.188222 16908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 13:59:42.214311 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 13:59:42.330300 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 13:59:42.361367 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 13:59:42.424862 17038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 13:59:42.425755 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 13:59:42.480243 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 13:59:42.969327 19437 tablet_copy_client.cc:523] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:42.972234 19437 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap starting.
I20240501 13:59:43.004840 19437 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:43.005316 19437 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap complete.
I20240501 13:59:43.005604 19437 ts_tablet_manager.cc:1367] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.009s
I20240501 13:59:43.006490 19437 raft_consensus.cc:348] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:43.006768 19437 raft_consensus.cc:729] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efb8dd25e14a46ae89eda6bbe5f940a2, State: Initialized, Role: LEARNER
I20240501 13:59:43.007007 19437 consensus_queue.cc:260] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:43.007948 19437 ts_tablet_manager.cc:1398] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 13:59:43.008847 17125 tablet_copy_service.cc:342] P 70ff9aaa9fa2414ea55225aafbd8fd89: Request end of tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-a3bfe952008f438c887a7d9574fc8bb9 received from {username='slave'} at 127.16.36.135:39913
I20240501 13:59:43.009029 17125 tablet_copy_service.cc:433] P 70ff9aaa9fa2414ea55225aafbd8fd89: ending tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-a3bfe952008f438c887a7d9574fc8bb9 on tablet a3bfe952008f438c887a7d9574fc8bb9 with peer efb8dd25e14a46ae89eda6bbe5f940a2
I20240501 13:59:43.040680 19443 tablet_copy_client.cc:523] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:43.044438 19443 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: Bootstrap starting.
I20240501 13:59:43.055073 19443 log.cc:824] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:43.093127 19443 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:43.093655 19443 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: Bootstrap complete.
I20240501 13:59:43.094120 19443 ts_tablet_manager.cc:1367] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: Time spent bootstrapping tablet: real 0.050s user 0.043s sys 0.004s
I20240501 13:59:43.099260 19443 raft_consensus.cc:348] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } }
I20240501 13:59:43.099716 19443 raft_consensus.cc:729] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e5565b8761548539de35f9318bec337, State: Initialized, Role: LEARNER
I20240501 13:59:43.100242 19443 consensus_queue.cc:260] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } }
I20240501 13:59:43.102031 19443 ts_tablet_manager.cc:1398] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20240501 13:59:43.103147 16995 tablet_copy_service.cc:342] P f4b54fe4701746c0930c9b0667c90519: Request end of tablet copy session 2e5565b8761548539de35f9318bec337-221688d4e2374a42b03e96d89166e0ee received from {username='slave'} at 127.16.36.136:44671
I20240501 13:59:43.103279 16995 tablet_copy_service.cc:433] P f4b54fe4701746c0930c9b0667c90519: ending tablet copy session 2e5565b8761548539de35f9318bec337-221688d4e2374a42b03e96d89166e0ee on tablet 221688d4e2374a42b03e96d89166e0ee with peer 2e5565b8761548539de35f9318bec337
I20240501 13:59:43.146417 18471 raft_consensus.cc:1203] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 13:59:43.320808 16557 catalog_manager.cc:4573] TS 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 101e3579d2954f2aac17d0f10ac84865: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:43.439932 18606 raft_consensus.cc:1203] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 13:59:43.535424 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 13:59:43.684288 19566 raft_consensus.cc:1050] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: attempting to promote NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 to VOTER
I20240501 13:59:43.685596 19566 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:43.688324 16975 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:43.688441 16845 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:43.689145 18471 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 LEARNER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:43.689150 19566 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:43.689585 19568 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:43.689919 19565 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:43.693244 19566 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:43.693990 18471 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:43.694089 16845 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:43.699062 16975 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:43.699431 16569 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 4 to 5, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:43.708241 17105 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:43.710871 16845 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:43.711648 19566 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:43.714869 19565 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:43.715301 18475 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:43.715881 16845 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:43.717008 19568 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:43.719822 16557 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3bfe952008f438c887a7d9574fc8bb9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:43.721081 18475 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:43.721637 16570 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 5 to 6, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:43.725705 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bfe952008f438c887a7d9574fc8bb9 with delete_type TABLET_DATA_TOMBSTONED (TS f4b54fe4701746c0930c9b0667c90519 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41412
I20240501 13:59:43.727205 19577 tablet_replica.cc:330] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: stopping tablet replica
I20240501 13:59:43.727449 19577 raft_consensus.cc:2229] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:43.727813 19577 raft_consensus.cc:2258] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:43.728855 19577 ts_tablet_manager.cc:1875] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:43.733594 19577 ts_tablet_manager.cc:1888] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 13:59:43.733724 19577 log.cc:1196] T a3bfe952008f438c887a7d9574fc8bb9 P f4b54fe4701746c0930c9b0667c90519: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3bfe952008f438c887a7d9574fc8bb9
I20240501 13:59:43.734486 16560 catalog_manager.cc:4830] TS f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163): tablet a3bfe952008f438c887a7d9574fc8bb9 (table rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]) successfully deleted
W20240501 13:59:43.794135 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 13:59:43.809430 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 13:59:43.853263 19589 raft_consensus.cc:1050] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: attempting to promote NON_VOTER 2e5565b8761548539de35f9318bec337 to VOTER
I20240501 13:59:43.854048 19589 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } }
I20240501 13:59:43.856778 17105 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:43.856824 18606 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:43.857460 19589 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:43.858017 16845 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:43.858373 19504 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:43.858739 19589 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:43.863615 19589 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.865027 16845 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.864929 18606 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.872170 16573 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 4 to 5, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.874258 17105 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.892026 16975 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } }
I20240501 13:59:43.896281 17105 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:43.897164 18606 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 13:59:43.897887 19589 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:43.898718 19589 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:43.901934 19588 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.902649 18606 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.907923 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 221688d4e2374a42b03e96d89166e0ee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:43.908179 17105 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.909222 16573 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 reported cstate change: config changed from index 5 to 6, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:43.914311 16824 tablet_service.cc:1514] Processing DeleteTablet for tablet 221688d4e2374a42b03e96d89166e0ee with delete_type TABLET_DATA_TOMBSTONED (TS d871186c2b234914ab63dc5e136f0f03 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42466
I20240501 13:59:43.915159 19600 tablet_replica.cc:330] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: stopping tablet replica
I20240501 13:59:43.915429 19600 raft_consensus.cc:2229] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:43.915784 19600 raft_consensus.cc:2258] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:43.916733 19600 ts_tablet_manager.cc:1875] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:43.923787 19600 ts_tablet_manager.cc:1888] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 13:59:43.923959 19600 log.cc:1196] T 221688d4e2374a42b03e96d89166e0ee P d871186c2b234914ab63dc5e136f0f03: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/221688d4e2374a42b03e96d89166e0ee
I20240501 13:59:43.924882 16560 catalog_manager.cc:4830] TS d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745): tablet 221688d4e2374a42b03e96d89166e0ee (table rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]) successfully deleted
W20240501 13:59:43.985069 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 13:59:44.144964 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 13:59:44.219837 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:44.232805 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:44.245899 16824 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:44.252094 18586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:44.252956 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:44.287966 16954 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:44.289258 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:44.381610 16845 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } }
I20240501 13:59:44.384521 17105 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 13:59:44.384981 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 13:59:44.385617 19601 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 13:59:44.385974 19652 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 13:59:44.390977 16780 consensus_peers.cc:485] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 -> Peer 2e5565b8761548539de35f9318bec337 (127.16.36.136:40683): Couldn't send request to peer 2e5565b8761548539de35f9318bec337. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f408fdebfb84082b0a55ad059e90694. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:44.396315 19652 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } } }
I20240501 13:59:44.397213 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } } }
I20240501 13:59:44.398397 17105 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } } }
I20240501 13:59:44.402355 16573 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 6 to 7, NON_VOTER 2e5565b8761548539de35f9318bec337 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:44.425200 16974 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:44.429226 18475 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 13:59:44.429297 17105 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 13:59:44.429890 19663 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 13:59:44.430318 19662 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 13:59:44.434413 16910 consensus_peers.cc:485] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 -> Peer 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959): Couldn't send request to peer 200e98827f7c44d29cd71ffd7faec1ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d6e800e4564a63b6a896b63519dca7. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:44.435166 19662 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:44.435984 18475 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:44.439975 17105 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:44.443816 16569 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 reported cstate change: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:44.458109 16845 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:44.460598 16974 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:44.461236 19424 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:44.461436 18214 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:44.462028 19424 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:44.464632 19652 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:44.465529 16974 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
W20240501 13:59:44.470839 16778 consensus_peers.cc:485] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 -> Peer efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399): Couldn't send request to peer efb8dd25e14a46ae89eda6bbe5f940a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 101e3579d2954f2aac17d0f10ac84865. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:44.470898 18214 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:44.471096 16573 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 8 to 9, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:44.753760 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 13:59:44.892066 19707 ts_tablet_manager.cc:902] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: Initiating tablet copy from peer f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:44.893575 19707 tablet_copy_client.cc:308] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:37163
I20240501 13:59:44.900619 16995 tablet_copy_service.cc:140] P f4b54fe4701746c0930c9b0667c90519: Received BeginTabletCopySession request for tablet d4d6e800e4564a63b6a896b63519dca7 from peer 200e98827f7c44d29cd71ffd7faec1ca ({username='slave'} at 127.16.36.134:49781)
I20240501 13:59:44.900827 16995 tablet_copy_service.cc:161] P f4b54fe4701746c0930c9b0667c90519: Beginning new tablet copy session on tablet d4d6e800e4564a63b6a896b63519dca7 from peer 200e98827f7c44d29cd71ffd7faec1ca at {username='slave'} at 127.16.36.134:49781: session id = 200e98827f7c44d29cd71ffd7faec1ca-d4d6e800e4564a63b6a896b63519dca7
I20240501 13:59:44.902868 16995 tablet_copy_source_session.cc:205] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.904247 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6e800e4564a63b6a896b63519dca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.909188 19707 tablet_copy_client.cc:783] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.909502 19707 tablet_copy_client.cc:647] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.911355 19711 tablet_copy_client.cc:1002] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Injecting latency into file download: 1500
W20240501 13:59:44.984004 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 13:59:45.006449 19728 ts_tablet_manager.cc:902] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: Initiating tablet copy from peer d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:45.007715 19728 tablet_copy_client.cc:308] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:42745
I20240501 13:59:45.013823 16865 tablet_copy_service.cc:140] P d871186c2b234914ab63dc5e136f0f03: Received BeginTabletCopySession request for tablet 5f408fdebfb84082b0a55ad059e90694 from peer 2e5565b8761548539de35f9318bec337 ({username='slave'} at 127.16.36.136:48553)
I20240501 13:59:45.014045 16865 tablet_copy_service.cc:161] P d871186c2b234914ab63dc5e136f0f03: Beginning new tablet copy session on tablet 5f408fdebfb84082b0a55ad059e90694 from peer 2e5565b8761548539de35f9318bec337 at {username='slave'} at 127.16.36.136:48553: session id = 2e5565b8761548539de35f9318bec337-5f408fdebfb84082b0a55ad059e90694
I20240501 13:59:45.016366 16865 tablet_copy_source_session.cc:205] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 13:59:45.016491 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 13:59:45.017846 19728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f408fdebfb84082b0a55ad059e90694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.022668 19728 tablet_copy_client.cc:783] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:45.022981 19728 tablet_copy_client.cc:647] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:45.024785 19732 tablet_copy_client.cc:1002] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:45.062613 19734 ts_tablet_manager.cc:902] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: Initiating tablet copy from peer d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 13:59:45.069530 19734 tablet_copy_client.cc:308] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:42745
I20240501 13:59:45.076886 16865 tablet_copy_service.cc:140] P d871186c2b234914ab63dc5e136f0f03: Received BeginTabletCopySession request for tablet 101e3579d2954f2aac17d0f10ac84865 from peer efb8dd25e14a46ae89eda6bbe5f940a2 ({username='slave'} at 127.16.36.135:34647)
I20240501 13:59:45.077091 16865 tablet_copy_service.cc:161] P d871186c2b234914ab63dc5e136f0f03: Beginning new tablet copy session on tablet 101e3579d2954f2aac17d0f10ac84865 from peer efb8dd25e14a46ae89eda6bbe5f940a2 at {username='slave'} at 127.16.36.135:34647: session id = efb8dd25e14a46ae89eda6bbe5f940a2-101e3579d2954f2aac17d0f10ac84865
I20240501 13:59:45.079378 16865 tablet_copy_source_session.cc:205] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:45.080636 19734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101e3579d2954f2aac17d0f10ac84865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:45.085340 19734 tablet_copy_client.cc:783] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:45.085639 19734 tablet_copy_client.cc:647] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:45.087738 19738 tablet_copy_client.cc:1002] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Injecting latency into file download: 1500
W20240501 13:59:45.188390 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 13:59:45.193653 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 13:59:45.847378 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 13:59:46.148217 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 13:59:46.204085 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 13:59:46.411798 19707 tablet_copy_client.cc:523] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:46.415145 19707 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap starting.
I20240501 13:59:46.453666 19707 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:46.454221 19707 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap complete.
I20240501 13:59:46.454486 19707 ts_tablet_manager.cc:1367] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.009s
I20240501 13:59:46.455449 19707 raft_consensus.cc:348] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:46.455708 19707 raft_consensus.cc:729] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 200e98827f7c44d29cd71ffd7faec1ca, State: Initialized, Role: LEARNER
I20240501 13:59:46.455904 19707 consensus_queue.cc:260] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:46.456678 19707 ts_tablet_manager.cc:1398] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca: Time spent starting tablet: real 0.002s user 0.001s sys 0.005s
I20240501 13:59:46.457674 16995 tablet_copy_service.cc:342] P f4b54fe4701746c0930c9b0667c90519: Request end of tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-d4d6e800e4564a63b6a896b63519dca7 received from {username='slave'} at 127.16.36.134:49781
I20240501 13:59:46.457888 16995 tablet_copy_service.cc:433] P f4b54fe4701746c0930c9b0667c90519: ending tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-d4d6e800e4564a63b6a896b63519dca7 on tablet d4d6e800e4564a63b6a896b63519dca7 with peer 200e98827f7c44d29cd71ffd7faec1ca
W20240501 13:59:46.488663 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 13:59:46.525223 19728 tablet_copy_client.cc:523] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:46.528466 19728 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: Bootstrap starting.
I20240501 13:59:46.567773 19728 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:46.568411 19728 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: Bootstrap complete.
I20240501 13:59:46.568729 19728 ts_tablet_manager.cc:1367] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.008s
I20240501 13:59:46.569906 19728 raft_consensus.cc:348] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } }
I20240501 13:59:46.570250 19728 raft_consensus.cc:729] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e5565b8761548539de35f9318bec337, State: Initialized, Role: LEARNER
I20240501 13:59:46.570513 19728 consensus_queue.cc:260] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: NON_VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: true } }
I20240501 13:59:46.571565 19728 ts_tablet_manager.cc:1398] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:46.572515 16865 tablet_copy_service.cc:342] P d871186c2b234914ab63dc5e136f0f03: Request end of tablet copy session 2e5565b8761548539de35f9318bec337-5f408fdebfb84082b0a55ad059e90694 received from {username='slave'} at 127.16.36.136:48553
I20240501 13:59:46.572695 16865 tablet_copy_service.cc:433] P d871186c2b234914ab63dc5e136f0f03: ending tablet copy session 2e5565b8761548539de35f9318bec337-5f408fdebfb84082b0a55ad059e90694 on tablet 5f408fdebfb84082b0a55ad059e90694 with peer 2e5565b8761548539de35f9318bec337
W20240501 13:59:46.585466 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 13:59:46.588176 19734 tablet_copy_client.cc:523] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:46.591440 19734 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap starting.
I20240501 13:59:46.635600 19734 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:46.636068 19734 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap complete.
I20240501 13:59:46.636346 19734 ts_tablet_manager.cc:1367] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent bootstrapping tablet: real 0.045s user 0.035s sys 0.008s
I20240501 13:59:46.637112 19734 raft_consensus.cc:348] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:46.637348 19734 raft_consensus.cc:729] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: efb8dd25e14a46ae89eda6bbe5f940a2, State: Initialized, Role: LEARNER
I20240501 13:59:46.637516 19734 consensus_queue.cc:260] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:46.638381 19734 ts_tablet_manager.cc:1398] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 13:59:46.639268 16865 tablet_copy_service.cc:342] P d871186c2b234914ab63dc5e136f0f03: Request end of tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-101e3579d2954f2aac17d0f10ac84865 received from {username='slave'} at 127.16.36.135:34647
I20240501 13:59:46.639436 16865 tablet_copy_service.cc:433] P d871186c2b234914ab63dc5e136f0f03: ending tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-101e3579d2954f2aac17d0f10ac84865 on tablet 101e3579d2954f2aac17d0f10ac84865 with peer efb8dd25e14a46ae89eda6bbe5f940a2
I20240501 13:59:46.813220 18345 raft_consensus.cc:1203] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 13:59:46.937321 18606 raft_consensus.cc:1203] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 13:59:47.155361 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 13:59:47.155965 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 13:59:47.174065 18475 raft_consensus.cc:1203] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240501 13:59:47.190336 16778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 13:59:47.227965 16908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 13:59:47.284019 19755 raft_consensus.cc:1050] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: attempting to promote NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca to VOTER
I20240501 13:59:47.285137 19755 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:47.292948 18475 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:47.292986 17105 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:47.294117 18345 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:47.294118 19896 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:47.299242 19897 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20240501 13:59:47.299607 19896 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:47.304350 19896 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.308246 17105 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.309492 18345 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.310691 18475 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.316239 16573 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca reported cstate change: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.328509 16974 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:47.330657 18345 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 13:59:47.331318 18475 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 13:59:47.331437 19908 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:47.332221 19897 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:47.334585 19908 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.335384 18345 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.337152 18475 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:47.339236 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4d6e800e4564a63b6a896b63519dca7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:47.341859 16570 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 reported cstate change: config changed from index 8 to 9, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:47.347785 17085 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6e800e4564a63b6a896b63519dca7 with delete_type TABLET_DATA_TOMBSTONED (TS 70ff9aaa9fa2414ea55225aafbd8fd89 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53578
I20240501 13:59:47.349099 19920 tablet_replica.cc:330] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: stopping tablet replica
I20240501 13:59:47.349581 19920 raft_consensus.cc:2229] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:47.350338 19920 raft_consensus.cc:2258] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:47.352550 19920 ts_tablet_manager.cc:1875] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:47.359160 19920 ts_tablet_manager.cc:1888] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 13:59:47.359352 19920 log.cc:1196] T d4d6e800e4564a63b6a896b63519dca7 P 70ff9aaa9fa2414ea55225aafbd8fd89: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4d6e800e4564a63b6a896b63519dca7
I20240501 13:59:47.360487 16557 catalog_manager.cc:4830] TS 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337): tablet d4d6e800e4564a63b6a896b63519dca7 (table rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]) successfully deleted
I20240501 13:59:47.399190 19938 raft_consensus.cc:1050] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: attempting to promote NON_VOTER 2e5565b8761548539de35f9318bec337 to VOTER
I20240501 13:59:47.400039 19938 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } }
I20240501 13:59:47.403380 17105 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:47.403460 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:47.403676 18606 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 1 LEARNER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:47.404064 19773 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:47.404527 19938 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:47.404870 19879 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:47.412145 19773 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:47.413162 17105 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:47.414752 18606 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:47.418924 16569 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 7 to 8, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:47.419740 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e5565b8761548539de35f9318bec337 (127.16.36.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
W20240501 13:59:47.440127 16557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 13:59:47.441857 16557 catalog_manager.cc:4573] TS 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 101e3579d2954f2aac17d0f10ac84865: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:47.618811 16844 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694"
dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46944
I20240501 13:59:47.619171 16844 raft_consensus.cc:595] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Received request to transfer leadership
W20240501 13:59:47.680088 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 13:59:47.781903 19773 raft_consensus.cc:1050] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03: attempting to promote NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 to VOTER
I20240501 13:59:47.782737 19773 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:47.785360 18214 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 13:59:47.785396 16974 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 13:59:47.786001 19938 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:47.786056 18475 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 13:59:47.786396 19879 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:47.787009 19773 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:47.790391 19773 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:47.791246 18475 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:47.794811 16974 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:47.795912 16565 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 9 to 10, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:47.798578 18214 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:47.801713 16844 consensus_queue.cc:237] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:47.805195 18475 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 13:59:47.806787 19879 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:47.808465 18213 raft_consensus.cc:1261] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Refusing update from remote peer d871186c2b234914ab63dc5e136f0f03: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 13:59:47.810007 19938 raft_consensus.cc:979] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: : Instructing follower 2e5565b8761548539de35f9318bec337 to start an election
I20240501 13:59:47.810215 19938 raft_consensus.cc:1067] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Signalling peer 2e5565b8761548539de35f9318bec337 to start an election
I20240501 13:59:47.810896 19773 consensus_queue.cc:1035] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:47.812188 18606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694"
dest_uuid: "2e5565b8761548539de35f9318bec337"
from {username='slave'} at 127.16.36.130:57999
I20240501 13:59:47.812531 18606 raft_consensus.cc:482] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:47.812744 18606 raft_consensus.cc:3037] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:47.816135 18606 raft_consensus.cc:504] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } }
I20240501 13:59:47.816758 19879 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:47.819043 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694" candidate_uuid: "2e5565b8761548539de35f9318bec337" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d871186c2b234914ab63dc5e136f0f03"
I20240501 13:59:47.819309 16844 raft_consensus.cc:3032] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:47.819478 16844 raft_consensus.cc:729] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d871186c2b234914ab63dc5e136f0f03, State: Running, Role: LEADER
I20240501 13:59:47.819260 18213 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:47.819850 16844 consensus_queue.cc:260] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } }
I20240501 13:59:47.820427 16844 raft_consensus.cc:3037] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Advancing to term 2
W20240501 13:59:47.827057 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 13:59:47.829254 16844 raft_consensus.cc:2454] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5565b8761548539de35f9318bec337 in term 2.
I20240501 13:59:47.830879 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 101e3579d2954f2aac17d0f10ac84865 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:47.831517 18475 raft_consensus.cc:2932] T 101e3579d2954f2aac17d0f10ac84865 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:47.833542 16570 catalog_manager.cc:5484] T 101e3579d2954f2aac17d0f10ac84865 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 10 to 11, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "d871186c2b234914ab63dc5e136f0f03" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:47.838189 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet 101e3579d2954f2aac17d0f10ac84865 with delete_type TABLET_DATA_TOMBSTONED (TS f4b54fe4701746c0930c9b0667c90519 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41412
I20240501 13:59:47.843173 18606 leader_election.cc:288] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [CANDIDATE]: Term 2 election: Requested vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745), 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959)
I20240501 13:59:47.863678 19971 tablet_replica.cc:330] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: stopping tablet replica
I20240501 13:59:47.864102 19971 raft_consensus.cc:2229] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:47.864607 19971 raft_consensus.cc:2258] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:47.865902 17105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694" candidate_uuid: "2e5565b8761548539de35f9318bec337" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:47.866189 17105 raft_consensus.cc:3037] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:47.866910 19971 ts_tablet_manager.cc:1875] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:47.869416 17105 raft_consensus.cc:2454] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5565b8761548539de35f9318bec337 in term 2.
I20240501 13:59:47.870474 18539 leader_election.cc:302] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2e5565b8761548539de35f9318bec337, 70ff9aaa9fa2414ea55225aafbd8fd89, d871186c2b234914ab63dc5e136f0f03; no voters:
I20240501 13:59:47.871373 19844 raft_consensus.cc:2781] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:47.871932 19844 raft_consensus.cc:686] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 2 LEADER]: Becoming Leader. State: Replica: 2e5565b8761548539de35f9318bec337, State: Running, Role: LEADER
I20240501 13:59:47.872593 19844 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } }
I20240501 13:59:47.873929 19971 ts_tablet_manager.cc:1888] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 13:59:47.874097 19971 log.cc:1196] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/101e3579d2954f2aac17d0f10ac84865
I20240501 13:59:47.876505 18345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f408fdebfb84082b0a55ad059e90694" candidate_uuid: "2e5565b8761548539de35f9318bec337" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "200e98827f7c44d29cd71ffd7faec1ca"
I20240501 13:59:47.876886 18345 raft_consensus.cc:3037] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:47.880604 18345 raft_consensus.cc:2454] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5565b8761548539de35f9318bec337 in term 2.
I20240501 13:59:47.882508 16570 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 reported cstate change: term changed from 1 to 2, leader changed from d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) to 2e5565b8761548539de35f9318bec337 (127.16.36.136). New cstate: current_term: 2 leader_uuid: "2e5565b8761548539de35f9318bec337" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:47.884137 16560 catalog_manager.cc:4830] TS f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163): tablet 101e3579d2954f2aac17d0f10ac84865 (table rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]) successfully deleted
I20240501 13:59:47.923075 16974 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:47.925684 18606 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 13:59:47.925863 16910 consensus_peers.cc:485] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 -> Peer 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959): Couldn't send request to peer 200e98827f7c44d29cd71ffd7faec1ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 221688d4e2374a42b03e96d89166e0ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:47.926141 17105 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 13:59:47.926307 19990 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 13:59:47.926777 19755 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 13:59:47.929649 19990 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:47.930670 17105 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:47.930859 18606 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:47.936461 16570 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 6 to 7, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
W20240501 13:59:47.955402 17038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 13:59:47.956837 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 13:59:48.013345 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 13:59:48.261260 17105 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:48.262042 19844 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:48.360126 20017 ts_tablet_manager.cc:902] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: Initiating tablet copy from peer f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:48.361618 20017 tablet_copy_client.cc:308] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:37163
I20240501 13:59:48.362390 16995 tablet_copy_service.cc:140] P f4b54fe4701746c0930c9b0667c90519: Received BeginTabletCopySession request for tablet 221688d4e2374a42b03e96d89166e0ee from peer 200e98827f7c44d29cd71ffd7faec1ca ({username='slave'} at 127.16.36.134:49781)
I20240501 13:59:48.362582 16995 tablet_copy_service.cc:161] P f4b54fe4701746c0930c9b0667c90519: Beginning new tablet copy session on tablet 221688d4e2374a42b03e96d89166e0ee from peer 200e98827f7c44d29cd71ffd7faec1ca at {username='slave'} at 127.16.36.134:49781: session id = 200e98827f7c44d29cd71ffd7faec1ca-221688d4e2374a42b03e96d89166e0ee
I20240501 13:59:48.365123 16995 tablet_copy_source_session.cc:205] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:48.366484 20017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221688d4e2374a42b03e96d89166e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:48.371685 20017 tablet_copy_client.cc:783] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:48.372054 20017 tablet_copy_client.cc:647] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:48.373930 20021 tablet_copy_client.cc:1002] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:48.437873 16844 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:48.438508 19844 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:48.445328 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:48.446313 19844 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 13:59:48.704167 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 13:59:48.729118 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 13:59:48.846719 18606 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } }
I20240501 13:59:48.849462 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 13:59:48.850188 20036 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:48.851725 17105 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 13:59:48.852809 20036 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:48.854166 20034 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:48.854943 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:48.856736 17105 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } }
I20240501 13:59:48.859539 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f408fdebfb84082b0a55ad059e90694 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:48.860527 16565 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 reported cstate change: config changed from index 8 to 10, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New cstate: current_term: 2 leader_uuid: "2e5565b8761548539de35f9318bec337" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:48.870074 16824 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f408fdebfb84082b0a55ad059e90694 with delete_type TABLET_DATA_TOMBSTONED (TS d871186c2b234914ab63dc5e136f0f03 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42466
I20240501 13:59:48.871616 20063 tablet_replica.cc:330] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: stopping tablet replica
I20240501 13:59:48.871997 20063 raft_consensus.cc:2229] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:48.872757 20063 raft_consensus.cc:2258] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:48.874224 20063 ts_tablet_manager.cc:1875] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:48.879951 20063 ts_tablet_manager.cc:1888] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 13:59:48.880172 20063 log.cc:1196] T 5f408fdebfb84082b0a55ad059e90694 P d871186c2b234914ab63dc5e136f0f03: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/5f408fdebfb84082b0a55ad059e90694
I20240501 13:59:48.881256 16560 catalog_manager.cc:4830] TS d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745): tablet 5f408fdebfb84082b0a55ad059e90694 (table rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]) successfully deleted
W20240501 13:59:48.894361 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 13:59:48.906416 17104 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:48.910151 18475 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 13:59:48.910727 16844 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 13:59:48.911109 19568 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 13:59:48.911484 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 13:59:48.911826 20075 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 13:59:48.916299 19568 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:48.918848 16844 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:48.922328 18475 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:48.925799 16569 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 reported cstate change: config changed from index 6 to 7, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
W20240501 13:59:48.927091 17038 consensus_peers.cc:485] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133:36049): Couldn't send request to peer 943a7d1d43144d56b35d62ebf72d1e25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3bfe952008f438c887a7d9574fc8bb9. This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:49.301963 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 13:59:49.521800 20112 ts_tablet_manager.cc:902] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: Initiating tablet copy from peer 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:49.523474 20112 tablet_copy_client.cc:308] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:37337
I20240501 13:59:49.530570 17125 tablet_copy_service.cc:140] P 70ff9aaa9fa2414ea55225aafbd8fd89: Received BeginTabletCopySession request for tablet a3bfe952008f438c887a7d9574fc8bb9 from peer 943a7d1d43144d56b35d62ebf72d1e25 ({username='slave'} at 127.16.36.133:36791)
I20240501 13:59:49.530807 17125 tablet_copy_service.cc:161] P 70ff9aaa9fa2414ea55225aafbd8fd89: Beginning new tablet copy session on tablet a3bfe952008f438c887a7d9574fc8bb9 from peer 943a7d1d43144d56b35d62ebf72d1e25 at {username='slave'} at 127.16.36.133:36791: session id = 943a7d1d43144d56b35d62ebf72d1e25-a3bfe952008f438c887a7d9574fc8bb9
I20240501 13:59:49.532948 17125 tablet_copy_source_session.cc:205] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:49.534317 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfe952008f438c887a7d9574fc8bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:49.539844 20112 tablet_copy_client.cc:783] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:49.540154 20112 tablet_copy_client.cc:647] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:49.541968 20116 tablet_copy_client.cc:1002] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:49.874845 20017 tablet_copy_client.cc:523] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:49.877938 20017 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap starting.
W20240501 13:59:49.888032 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 13:59:49.917490 20017 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:49.917969 20017 tablet_bootstrap.cc:492] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap complete.
I20240501 13:59:49.918275 20017 ts_tablet_manager.cc:1367] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.012s
I20240501 13:59:49.919114 20017 raft_consensus.cc:348] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:49.919364 20017 raft_consensus.cc:729] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 200e98827f7c44d29cd71ffd7faec1ca, State: Initialized, Role: LEARNER
I20240501 13:59:49.919592 20017 consensus_queue.cc:260] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:49.920565 20017 ts_tablet_manager.cc:1398] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 13:59:49.921599 16995 tablet_copy_service.cc:342] P f4b54fe4701746c0930c9b0667c90519: Request end of tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-221688d4e2374a42b03e96d89166e0ee received from {username='slave'} at 127.16.36.134:49781
I20240501 13:59:49.921777 16995 tablet_copy_service.cc:433] P f4b54fe4701746c0930c9b0667c90519: ending tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-221688d4e2374a42b03e96d89166e0ee on tablet 221688d4e2374a42b03e96d89166e0ee with peer 200e98827f7c44d29cd71ffd7faec1ca
I20240501 13:59:49.926745 18345 raft_consensus.cc:1203] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 13:59:50.225749 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 13:59:50.259590 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 13:59:50.271816 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 13:59:50.531409 20175 raft_consensus.cc:1050] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: attempting to promote NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca to VOTER
I20240501 13:59:50.532326 20175 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:50.535184 17104 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:50.535305 18606 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:50.535367 18345 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:50.535931 20177 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:50.536428 20175 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:50.536846 20091 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:50.540532 20177 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:50.541293 18606 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:50.544440 18345 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:50.545421 17104 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:50.546667 16570 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 reported cstate change: config changed from index 7 to 8, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 13:59:50.676476 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 13:59:50.760269 16974 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee"
dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43166
I20240501 13:59:50.760617 16974 raft_consensus.cc:595] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:50.967875 20177 raft_consensus.cc:979] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: : Instructing follower 70ff9aaa9fa2414ea55225aafbd8fd89 to start an election
I20240501 13:59:50.968144 20175 raft_consensus.cc:1067] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Signalling peer 70ff9aaa9fa2414ea55225aafbd8fd89 to start an election
I20240501 13:59:50.969215 17104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee"
dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
from {username='slave'} at 127.16.36.131:50099
I20240501 13:59:50.969561 17104 raft_consensus.cc:482] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:50.969732 17104 raft_consensus.cc:3037] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:50.972838 17104 raft_consensus.cc:504] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:50.974071 17104 leader_election.cc:288] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 election: Requested vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), 2e5565b8761548539de35f9318bec337 (127.16.36.136:40683), 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959)
I20240501 13:59:50.974885 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:50.975229 16974 raft_consensus.cc:3032] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:50.975354 16974 raft_consensus.cc:729] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:50.975792 16974 consensus_queue.cc:260] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:50.976637 16974 raft_consensus.cc:3037] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:50.982678 16974 raft_consensus.cc:2454] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 2.
I20240501 13:59:50.998126 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2e5565b8761548539de35f9318bec337"
I20240501 13:59:50.998579 18606 raft_consensus.cc:3037] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:50.998528 18345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221688d4e2374a42b03e96d89166e0ee" candidate_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "200e98827f7c44d29cd71ffd7faec1ca"
I20240501 13:59:50.998852 18345 raft_consensus.cc:3037] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:51.001550 18345 raft_consensus.cc:2454] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 2.
I20240501 13:59:51.002056 17040 leader_election.cc:302] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 200e98827f7c44d29cd71ffd7faec1ca, 70ff9aaa9fa2414ea55225aafbd8fd89, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:51.002504 20187 raft_consensus.cc:2781] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:51.002743 20187 raft_consensus.cc:686] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 LEADER]: Becoming Leader. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Running, Role: LEADER
I20240501 13:59:51.003269 18606 raft_consensus.cc:2454] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70ff9aaa9fa2414ea55225aafbd8fd89 in term 2.
I20240501 13:59:51.003134 20187 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:51.007043 16569 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: term changed from 1 to 2, leader changed from f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) to 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132). New cstate: current_term: 2 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:51.042459 20112 tablet_copy_client.cc:523] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:51.045728 20112 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap starting.
I20240501 13:59:51.086867 20112 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:51.087446 20112 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap complete.
I20240501 13:59:51.087762 20112 ts_tablet_manager.cc:1367] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: Time spent bootstrapping tablet: real 0.042s user 0.041s sys 0.002s
I20240501 13:59:51.088794 20112 raft_consensus.cc:348] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:51.089066 20112 raft_consensus.cc:729] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 943a7d1d43144d56b35d62ebf72d1e25, State: Initialized, Role: LEARNER
I20240501 13:59:51.089296 20112 consensus_queue.cc:260] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:51.090216 20112 ts_tablet_manager.cc:1398] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:51.091156 17125 tablet_copy_service.cc:342] P 70ff9aaa9fa2414ea55225aafbd8fd89: Request end of tablet copy session 943a7d1d43144d56b35d62ebf72d1e25-a3bfe952008f438c887a7d9574fc8bb9 received from {username='slave'} at 127.16.36.133:36791
I20240501 13:59:51.091297 17125 tablet_copy_service.cc:433] P 70ff9aaa9fa2414ea55225aafbd8fd89: ending tablet copy session 943a7d1d43144d56b35d62ebf72d1e25-a3bfe952008f438c887a7d9574fc8bb9 on tablet a3bfe952008f438c887a7d9574fc8bb9 with peer 943a7d1d43144d56b35d62ebf72d1e25
W20240501 13:59:51.139966 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 13:59:51.269562 18213 raft_consensus.cc:1203] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 13:59:51.280535 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 13:59:51.383466 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 13:59:51.422461 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 13:59:51.436216 16974 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:51.436964 20186 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:51.466419 18345 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:51.467229 20186 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:51.471870 18606 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 2 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 13:59:51.472672 20186 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 13:59:51.678190 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 13:59:51.809271 20186 raft_consensus.cc:1050] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: attempting to promote NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 to VOTER
I20240501 13:59:51.810338 20186 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:51.813086 18213 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:51.813249 18471 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:51.813755 16844 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 13:59:51.813920 20187 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:51.814646 20186 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 13:59:51.816027 20187 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 13:59:51.819689 20187 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.823794 16844 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.824535 18213 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.828403 16573 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 7 to 8, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:51.851222 18471 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } attrs { replace: true } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.856248 17104 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:51.858662 18213 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 13:59:51.859074 18471 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 13:59:51.859341 20258 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:51.859681 20187 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 13:59:51.861996 20186 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.862637 18471 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.862915 18213 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.866683 16557 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3bfe952008f438c887a7d9574fc8bb9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:51.868002 16565 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 reported cstate change: config changed from index 8 to 9, VOTER d871186c2b234914ab63dc5e136f0f03 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:51.872052 16824 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bfe952008f438c887a7d9574fc8bb9 with delete_type TABLET_DATA_TOMBSTONED (TS d871186c2b234914ab63dc5e136f0f03 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42466
I20240501 13:59:51.873013 20267 tablet_replica.cc:330] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: stopping tablet replica
I20240501 13:59:51.873277 20267 raft_consensus.cc:2229] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:51.873942 20267 raft_consensus.cc:2258] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:51.874956 20267 ts_tablet_manager.cc:1875] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:51.880225 20267 ts_tablet_manager.cc:1888] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 13:59:51.880692 20267 log.cc:1196] T a3bfe952008f438c887a7d9574fc8bb9 P d871186c2b234914ab63dc5e136f0f03: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3bfe952008f438c887a7d9574fc8bb9
I20240501 13:59:51.881641 16560 catalog_manager.cc:4830] TS d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745): tablet a3bfe952008f438c887a7d9574fc8bb9 (table rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]) successfully deleted
I20240501 13:59:52.049383 17104 consensus_queue.cc:237] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:52.051335 18345 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 13:59:52.051386 18606 raft_consensus.cc:1261] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 2 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 13:59:52.052018 20186 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:52.052371 20187 consensus_queue.cc:1035] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:52.054914 20258 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:52.058977 16557 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 221688d4e2374a42b03e96d89166e0ee with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:52.059032 18606 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 2e5565b8761548539de35f9318bec337 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:52.059661 18345 raft_consensus.cc:2932] T 221688d4e2374a42b03e96d89166e0ee P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:52.060009 16573 catalog_manager.cc:5484] T 221688d4e2374a42b03e96d89166e0ee P 70ff9aaa9fa2414ea55225aafbd8fd89 reported cstate change: config changed from index 8 to 10, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:52.066459 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet 221688d4e2374a42b03e96d89166e0ee with delete_type TABLET_DATA_TOMBSTONED (TS f4b54fe4701746c0930c9b0667c90519 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41412
I20240501 13:59:52.075294 20283 tablet_replica.cc:330] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: stopping tablet replica
I20240501 13:59:52.075534 20283 raft_consensus.cc:2229] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:52.075776 20283 raft_consensus.cc:2258] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:52.076571 20283 ts_tablet_manager.cc:1875] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:52.081828 20283 ts_tablet_manager.cc:1888] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 13:59:52.081965 20283 log.cc:1196] T 221688d4e2374a42b03e96d89166e0ee P f4b54fe4701746c0930c9b0667c90519: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/221688d4e2374a42b03e96d89166e0ee
I20240501 13:59:52.082850 16560 catalog_manager.cc:4830] TS f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163): tablet 221688d4e2374a42b03e96d89166e0ee (table rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]) successfully deleted
W20240501 13:59:52.236299 16908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 13:59:52.250600 18325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:52.264612 18586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:52.277856 16824 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 13:59:52.298646 16778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 13:59:52.308337 18194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:52.310514 17085 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:52.347952 16954 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 13:59:52.348539 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:52.425326 16974 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:52.428570 18345 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 13:59:52.429280 18471 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 13:59:52.429291 20177 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:52.430114 20177 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:52.433859 18345 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:52.433494 20336 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:52.434255 18471 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
I20240501 13:59:52.440953 16572 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca reported cstate change: config changed from index 9 to 10, NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } } }
W20240501 13:59:52.451295 16908 consensus_peers.cc:485] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 -> Peer 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133:36049): Couldn't send request to peer 943a7d1d43144d56b35d62ebf72d1e25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d6e800e4564a63b6a896b63519dca7. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:52.459735 18606 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:52.462390 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 13:59:52.463227 19844 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 13:59:52.470273 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 13:59:52.471148 19844 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:52.476022 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
I20240501 13:59:52.477669 17104 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 13:59:52.478469 20349 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:52.483485 16565 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca reported cstate change: config changed from index 10 to 11, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New cstate: current_term: 2 leader_uuid: "2e5565b8761548539de35f9318bec337" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
W20240501 13:59:52.489732 18539 consensus_peers.cc:485] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 -> Peer efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399): Couldn't send request to peer efb8dd25e14a46ae89eda6bbe5f940a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f408fdebfb84082b0a55ad059e90694. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:52.489344 17104 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } } }
W20240501 13:59:52.608702 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 13:59:52.807413 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 13:59:52.898196 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 13:59:52.994290 20386 ts_tablet_manager.cc:902] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: Initiating tablet copy from peer f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163)
I20240501 13:59:52.995518 20386 tablet_copy_client.cc:308] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:37163
I20240501 13:59:53.005300 16995 tablet_copy_service.cc:140] P f4b54fe4701746c0930c9b0667c90519: Received BeginTabletCopySession request for tablet d4d6e800e4564a63b6a896b63519dca7 from peer 943a7d1d43144d56b35d62ebf72d1e25 ({username='slave'} at 127.16.36.133:40603)
I20240501 13:59:53.005487 16995 tablet_copy_service.cc:161] P f4b54fe4701746c0930c9b0667c90519: Beginning new tablet copy session on tablet d4d6e800e4564a63b6a896b63519dca7 from peer 943a7d1d43144d56b35d62ebf72d1e25 at {username='slave'} at 127.16.36.133:40603: session id = 943a7d1d43144d56b35d62ebf72d1e25-d4d6e800e4564a63b6a896b63519dca7
I20240501 13:59:53.007786 16995 tablet_copy_source_session.cc:205] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:53.009207 20386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6e800e4564a63b6a896b63519dca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:53.014318 20386 tablet_copy_client.cc:783] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:53.014654 20386 tablet_copy_client.cc:647] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:53.016489 20390 tablet_copy_client.cc:1002] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:53.055446 20392 ts_tablet_manager.cc:902] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: Initiating tablet copy from peer 2e5565b8761548539de35f9318bec337 (127.16.36.136:40683)
I20240501 13:59:53.057255 20392 tablet_copy_client.cc:308] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.136:40683
I20240501 13:59:53.066669 18626 tablet_copy_service.cc:140] P 2e5565b8761548539de35f9318bec337: Received BeginTabletCopySession request for tablet 5f408fdebfb84082b0a55ad059e90694 from peer efb8dd25e14a46ae89eda6bbe5f940a2 ({username='slave'} at 127.16.36.135:43231)
I20240501 13:59:53.066913 18626 tablet_copy_service.cc:161] P 2e5565b8761548539de35f9318bec337: Beginning new tablet copy session on tablet 5f408fdebfb84082b0a55ad059e90694 from peer efb8dd25e14a46ae89eda6bbe5f940a2 at {username='slave'} at 127.16.36.135:43231: session id = efb8dd25e14a46ae89eda6bbe5f940a2-5f408fdebfb84082b0a55ad059e90694
I20240501 13:59:53.069831 18626 tablet_copy_source_session.cc:205] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:53.071995 20392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f408fdebfb84082b0a55ad059e90694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:53.078608 20392 tablet_copy_client.cc:783] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:53.079073 20392 tablet_copy_client.cc:647] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:53.081624 20396 tablet_copy_client.cc:1002] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Injecting latency into file download: 1500
W20240501 13:59:53.289160 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 13:59:53.408591 17038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 13:59:53.731220 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 13:59:53.811128 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 13:59:53.822194 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 13:59:53.968498 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 13:59:54.340690 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 13:59:54.516960 20386 tablet_copy_client.cc:523] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:54.520040 20386 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap starting.
I20240501 13:59:54.561388 20386 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:54.561952 20386 tablet_bootstrap.cc:492] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: Bootstrap complete.
I20240501 13:59:54.562274 20386 ts_tablet_manager.cc:1367] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: Time spent bootstrapping tablet: real 0.042s user 0.030s sys 0.009s
I20240501 13:59:54.563236 20386 raft_consensus.cc:348] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:54.563567 20386 raft_consensus.cc:729] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 943a7d1d43144d56b35d62ebf72d1e25, State: Initialized, Role: LEARNER
I20240501 13:59:54.563807 20386 consensus_queue.cc:260] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 13:59:54.564884 20386 ts_tablet_manager.cc:1398] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25: Time spent starting tablet: real 0.002s user 0.002s sys 0.004s
I20240501 13:59:54.565920 16995 tablet_copy_service.cc:342] P f4b54fe4701746c0930c9b0667c90519: Request end of tablet copy session 943a7d1d43144d56b35d62ebf72d1e25-d4d6e800e4564a63b6a896b63519dca7 received from {username='slave'} at 127.16.36.133:40603
I20240501 13:59:54.566094 16995 tablet_copy_service.cc:433] P f4b54fe4701746c0930c9b0667c90519: ending tablet copy session 943a7d1d43144d56b35d62ebf72d1e25-d4d6e800e4564a63b6a896b63519dca7 on tablet d4d6e800e4564a63b6a896b63519dca7 with peer 943a7d1d43144d56b35d62ebf72d1e25
I20240501 13:59:54.582182 20392 tablet_copy_client.cc:523] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:54.585527 20392 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap starting.
I20240501 13:59:54.621837 20392 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:54.622380 20392 tablet_bootstrap.cc:492] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: Bootstrap complete.
I20240501 13:59:54.622716 20392 ts_tablet_manager.cc:1367] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent bootstrapping tablet: real 0.037s user 0.032s sys 0.008s
I20240501 13:59:54.623670 20392 raft_consensus.cc:348] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:54.624030 20392 raft_consensus.cc:729] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: efb8dd25e14a46ae89eda6bbe5f940a2, State: Initialized, Role: LEARNER
I20240501 13:59:54.624336 20392 consensus_queue.cc:260] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: NON_VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: true } }
I20240501 13:59:54.625265 20392 ts_tablet_manager.cc:1398] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:54.626274 18626 tablet_copy_service.cc:342] P 2e5565b8761548539de35f9318bec337: Request end of tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-5f408fdebfb84082b0a55ad059e90694 received from {username='slave'} at 127.16.36.135:43231
I20240501 13:59:54.626502 18626 tablet_copy_service.cc:433] P 2e5565b8761548539de35f9318bec337: ending tablet copy session efb8dd25e14a46ae89eda6bbe5f940a2-5f408fdebfb84082b0a55ad059e90694 on tablet 5f408fdebfb84082b0a55ad059e90694 with peer efb8dd25e14a46ae89eda6bbe5f940a2
I20240501 13:59:54.726923 18471 raft_consensus.cc:1203] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240501 13:59:55.057415 18213 raft_consensus.cc:1203] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 13:59:55.117794 20507 raft_consensus.cc:1050] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337: attempting to promote NON_VOTER efb8dd25e14a46ae89eda6bbe5f940a2 to VOTER
I20240501 13:59:55.119134 20507 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:55.121986 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:55.122274 17104 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:55.122555 18471 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 LEARNER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:55.122778 20506 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:55.123243 20507 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:55.123699 20509 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:55.129048 18471 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.129606 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.128410 20507 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.135430 16570 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 reported cstate change: config changed from index 11 to 12, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2e5565b8761548539de35f9318bec337" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.137321 17104 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.144767 18606 consensus_queue.cc:237] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } }
I20240501 13:59:55.147213 18471 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 13:59:55.147835 18345 raft_consensus.cc:1261] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 2e5565b8761548539de35f9318bec337: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 13:59:55.147886 20506 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 13:59:55.148408 20458 consensus_queue.cc:1035] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 13:59:55.150466 20506 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.151230 18471 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.151939 18345 raft_consensus.cc:2932] T 5f408fdebfb84082b0a55ad059e90694 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.155368 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f408fdebfb84082b0a55ad059e90694 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:55.157009 16570 catalog_manager.cc:5484] T 5f408fdebfb84082b0a55ad059e90694 P efb8dd25e14a46ae89eda6bbe5f940a2 reported cstate change: config changed from index 12 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New cstate: current_term: 2 leader_uuid: "2e5565b8761548539de35f9318bec337" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "2e5565b8761548539de35f9318bec337" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 40683 } attrs { promote: false } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } }
I20240501 13:59:55.161420 17085 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f408fdebfb84082b0a55ad059e90694 with delete_type TABLET_DATA_TOMBSTONED (TS 70ff9aaa9fa2414ea55225aafbd8fd89 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:53578
I20240501 13:59:55.162420 20518 tablet_replica.cc:330] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: stopping tablet replica
I20240501 13:59:55.162689 20518 raft_consensus.cc:2229] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:55.163022 20518 raft_consensus.cc:2258] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.163983 20518 ts_tablet_manager.cc:1875] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:55.169178 20518 ts_tablet_manager.cc:1888] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 13:59:55.169438 20518 log.cc:1196] T 5f408fdebfb84082b0a55ad059e90694 P 70ff9aaa9fa2414ea55225aafbd8fd89: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f408fdebfb84082b0a55ad059e90694
I20240501 13:59:55.170369 16557 catalog_manager.cc:4830] TS 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337): tablet 5f408fdebfb84082b0a55ad059e90694 (table rebalance_test_table_3 [id=9118eded22854673bb91b04e1e0bd660]) successfully deleted
W20240501 13:59:55.180927 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 13:59:55.210532 17104 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
W20240501 13:59:55.213851 17040 consensus_peers.cc:485] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959): Couldn't send request to peer 200e98827f7c44d29cd71ffd7faec1ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3bfe952008f438c887a7d9574fc8bb9. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:55.213894 18213 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 13:59:55.214242 18471 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 13:59:55.214550 20536 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:55.215015 20241 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 13:59:55.218387 20241 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:55.219069 18471 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:55.219489 18213 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
I20240501 13:59:55.225395 16565 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 reported cstate change: config changed from index 9 to 10, NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } } }
W20240501 13:59:55.364918 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 13:59:55.409065 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 13:59:55.438153 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 13:59:55.576555 20562 raft_consensus.cc:1050] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: attempting to promote NON_VOTER 943a7d1d43144d56b35d62ebf72d1e25 to VOTER
I20240501 13:59:55.577342 20562 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:55.580585 18471 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 13:59:55.580677 18345 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 13:59:55.581390 20563 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:55.581971 18213 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 LEARNER]: Refusing update from remote peer f4b54fe4701746c0930c9b0667c90519: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 13:59:55.581825 20565 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:55.582595 20562 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:55.586580 20563 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:55.587280 18213 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:55.587509 18345 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:55.593732 18471 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:55.593914 16569 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca reported cstate change: config changed from index 10 to 11, 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f4b54fe4701746c0930c9b0667c90519" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
W20240501 13:59:55.648700 16557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
W20240501 13:59:55.650178 16557 catalog_manager.cc:4573] TS 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 101e3579d2954f2aac17d0f10ac84865: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111)
I20240501 13:59:55.655826 20571 ts_tablet_manager.cc:902] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: Initiating tablet copy from peer 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337)
I20240501 13:59:55.657714 20571 tablet_copy_client.cc:308] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:37337
I20240501 13:59:55.666709 17125 tablet_copy_service.cc:140] P 70ff9aaa9fa2414ea55225aafbd8fd89: Received BeginTabletCopySession request for tablet a3bfe952008f438c887a7d9574fc8bb9 from peer 200e98827f7c44d29cd71ffd7faec1ca ({username='slave'} at 127.16.36.134:35529)
I20240501 13:59:55.666939 17125 tablet_copy_service.cc:161] P 70ff9aaa9fa2414ea55225aafbd8fd89: Beginning new tablet copy session on tablet a3bfe952008f438c887a7d9574fc8bb9 from peer 200e98827f7c44d29cd71ffd7faec1ca at {username='slave'} at 127.16.36.134:35529: session id = 200e98827f7c44d29cd71ffd7faec1ca-a3bfe952008f438c887a7d9574fc8bb9
I20240501 13:59:55.669328 17125 tablet_copy_source_session.cc:205] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:55.671032 20571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfe952008f438c887a7d9574fc8bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:55.676836 20571 tablet_copy_client.cc:783] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:55.677150 20571 tablet_copy_client.cc:647] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:55.679733 20575 tablet_copy_client.cc:1002] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Injecting latency into file download: 1500
I20240501 13:59:55.709131 16974 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7"
dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59860
I20240501 13:59:55.709424 16974 raft_consensus.cc:595] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Received request to transfer leadership
W20240501 13:59:55.804231 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 13:59:55.941475 16974 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7"
dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59876
I20240501 13:59:55.941766 16974 raft_consensus.cc:595] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:55.976680 20565 raft_consensus.cc:979] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: : Instructing follower 200e98827f7c44d29cd71ffd7faec1ca to start an election
I20240501 13:59:55.976887 20565 raft_consensus.cc:1067] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Signalling peer 200e98827f7c44d29cd71ffd7faec1ca to start an election
I20240501 13:59:55.977733 18345 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7"
dest_uuid: "200e98827f7c44d29cd71ffd7faec1ca"
from {username='slave'} at 127.16.36.131:42927
I20240501 13:59:55.978042 18345 raft_consensus.cc:482] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:55.978163 18345 raft_consensus.cc:3037] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:55.980437 18345 raft_consensus.cc:504] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:55.982197 18345 leader_election.cc:288] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [CANDIDATE]: Term 2 election: Requested vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399), 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133:36049)
I20240501 13:59:55.982743 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7" candidate_uuid: "200e98827f7c44d29cd71ffd7faec1ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "f4b54fe4701746c0930c9b0667c90519"
I20240501 13:59:55.983007 16974 raft_consensus.cc:3032] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:55.983100 16974 raft_consensus.cc:729] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f4b54fe4701746c0930c9b0667c90519, State: Running, Role: LEADER
I20240501 13:59:55.983397 16974 consensus_queue.cc:260] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:55.984019 16974 raft_consensus.cc:3037] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:55.988310 16974 raft_consensus.cc:2454] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200e98827f7c44d29cd71ffd7faec1ca in term 2.
I20240501 13:59:55.998171 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7" candidate_uuid: "200e98827f7c44d29cd71ffd7faec1ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2"
I20240501 13:59:55.998567 18471 raft_consensus.cc:3037] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:55.999469 18213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6e800e4564a63b6a896b63519dca7" candidate_uuid: "200e98827f7c44d29cd71ffd7faec1ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "943a7d1d43144d56b35d62ebf72d1e25"
I20240501 13:59:55.999797 18213 raft_consensus.cc:3037] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:56.001510 18471 raft_consensus.cc:2454] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200e98827f7c44d29cd71ffd7faec1ca in term 2.
I20240501 13:59:56.002140 18278 leader_election.cc:302] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 200e98827f7c44d29cd71ffd7faec1ca, efb8dd25e14a46ae89eda6bbe5f940a2, f4b54fe4701746c0930c9b0667c90519; no voters:
I20240501 13:59:56.002379 18213 raft_consensus.cc:2454] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200e98827f7c44d29cd71ffd7faec1ca in term 2.
I20240501 13:59:56.002802 20513 raft_consensus.cc:2781] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:56.003566 20513 raft_consensus.cc:686] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 LEADER]: Becoming Leader. State: Replica: 200e98827f7c44d29cd71ffd7faec1ca, State: Running, Role: LEADER
I20240501 13:59:56.004189 20513 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:56.008037 16565 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca reported cstate change: term changed from 1 to 2, leader changed from f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) to 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134). New cstate: current_term: 2 leader_uuid: "200e98827f7c44d29cd71ffd7faec1ca" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:56.168079 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 13:59:56.271101 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 13:59:56.330811 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 13:59:56.369593 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 13:59:56.427804 18213 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Refusing update from remote peer 200e98827f7c44d29cd71ffd7faec1ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:56.428632 20513 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:56.600489 18471 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Refusing update from remote peer 200e98827f7c44d29cd71ffd7faec1ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:56.601382 20513 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:56.627907 16974 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Refusing update from remote peer 200e98827f7c44d29cd71ffd7faec1ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:56.628742 20651 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:56.646240 18345 consensus_queue.cc:237] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } }
I20240501 13:59:56.648417 18471 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Refusing update from remote peer 200e98827f7c44d29cd71ffd7faec1ca: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 13:59:56.648784 18213 raft_consensus.cc:1261] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Refusing update from remote peer 200e98827f7c44d29cd71ffd7faec1ca: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 13:59:56.649037 20652 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 13:59:56.649437 20651 consensus_queue.cc:1035] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 13:59:56.651495 20652 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:56.652364 18471 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:56.653710 18213 raft_consensus.cc:2932] T d4d6e800e4564a63b6a896b63519dca7 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:56.656800 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4d6e800e4564a63b6a896b63519dca7 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:56.657508 16572 catalog_manager.cc:5484] T d4d6e800e4564a63b6a896b63519dca7 P efb8dd25e14a46ae89eda6bbe5f940a2 reported cstate change: config changed from index 11 to 13, VOTER f4b54fe4701746c0930c9b0667c90519 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "200e98827f7c44d29cd71ffd7faec1ca" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } }
I20240501 13:59:56.663435 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6e800e4564a63b6a896b63519dca7 with delete_type TABLET_DATA_TOMBSTONED (TS f4b54fe4701746c0930c9b0667c90519 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:41412
I20240501 13:59:56.664327 20661 tablet_replica.cc:330] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: stopping tablet replica
I20240501 13:59:56.664575 20661 raft_consensus.cc:2229] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:56.665517 20661 raft_consensus.cc:2258] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:56.666399 20661 ts_tablet_manager.cc:1875] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:56.671365 20661 ts_tablet_manager.cc:1888] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 13:59:56.671500 20661 log.cc:1196] T d4d6e800e4564a63b6a896b63519dca7 P f4b54fe4701746c0930c9b0667c90519: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/d4d6e800e4564a63b6a896b63519dca7
I20240501 13:59:56.672298 16560 catalog_manager.cc:4830] TS f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163): tablet d4d6e800e4564a63b6a896b63519dca7 (table rebalance_test_table_4 [id=947fe07ea5964a77a7f950cd2d35a1a8]) successfully deleted
W20240501 13:59:57.144270 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 13:59:57.180249 20571 tablet_copy_client.cc:523] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:57.183493 20571 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap starting.
I20240501 13:59:57.220432 20571 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:57.220971 20571 tablet_bootstrap.cc:492] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: Bootstrap complete.
I20240501 13:59:57.221288 20571 ts_tablet_manager.cc:1367] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.009s
I20240501 13:59:57.222010 20571 raft_consensus.cc:348] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:57.222276 20571 raft_consensus.cc:729] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 200e98827f7c44d29cd71ffd7faec1ca, State: Initialized, Role: LEARNER
I20240501 13:59:57.222487 20571 consensus_queue.cc:260] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: NON_VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: true } }
I20240501 13:59:57.223080 20571 ts_tablet_manager.cc:1398] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 13:59:57.223928 17125 tablet_copy_service.cc:342] P 70ff9aaa9fa2414ea55225aafbd8fd89: Request end of tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-a3bfe952008f438c887a7d9574fc8bb9 received from {username='slave'} at 127.16.36.134:35529
I20240501 13:59:57.224071 17125 tablet_copy_service.cc:433] P 70ff9aaa9fa2414ea55225aafbd8fd89: ending tablet copy session 200e98827f7c44d29cd71ffd7faec1ca-a3bfe952008f438c887a7d9574fc8bb9 on tablet a3bfe952008f438c887a7d9574fc8bb9 with peer 200e98827f7c44d29cd71ffd7faec1ca
W20240501 13:59:57.314052 16778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 13:59:57.358969 16908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 13:59:57.629542 18345 raft_consensus.cc:1203] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 13:59:57.655048 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 13:59:57.790841 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 13:59:57.799685 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 13:59:58.007159 20710 raft_consensus.cc:1050] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: attempting to promote NON_VOTER 200e98827f7c44d29cd71ffd7faec1ca to VOTER
I20240501 13:59:58.008278 20710 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:58.011616 18471 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 13:59:58.011636 18213 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 13:59:58.012039 18345 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 LEARNER]: Refusing update from remote peer 70ff9aaa9fa2414ea55225aafbd8fd89: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 13:59:58.012410 20712 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:58.012912 20710 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:58.013289 20709 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 13:59:58.016703 20712 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:58.017432 18213 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:58.021037 18345 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:58.021180 18471 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:58.023394 16565 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 reported cstate change: config changed from index 10 to 11, 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
W20240501 13:59:58.036911 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 13:59:58.133119 16778 consensus_peers.cc:485] T 8f27c70cad684568af8d8119559475cc P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 13:59:58.147318 17104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9"
dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35314
I20240501 13:59:58.147598 17104 raft_consensus.cc:595] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:58.369114 17104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9"
dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35322
I20240501 13:59:58.369369 17104 raft_consensus.cc:595] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:58.408344 20710 raft_consensus.cc:979] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: : Instructing follower 943a7d1d43144d56b35d62ebf72d1e25 to start an election
I20240501 13:59:58.408579 20663 raft_consensus.cc:1067] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Signalling peer 943a7d1d43144d56b35d62ebf72d1e25 to start an election
I20240501 13:59:58.409466 18213 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9"
dest_uuid: "943a7d1d43144d56b35d62ebf72d1e25"
from {username='slave'} at 127.16.36.132:56963
I20240501 13:59:58.409745 18213 raft_consensus.cc:482] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:58.409893 18213 raft_consensus.cc:3037] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:58.412295 18213 raft_consensus.cc:504] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:58.414119 18213 leader_election.cc:288] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [CANDIDATE]: Term 2 election: Requested vote from peers 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337), efb8dd25e14a46ae89eda6bbe5f940a2 (127.16.36.135:39399), 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959)
I20240501 13:59:58.414485 17104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9" candidate_uuid: "943a7d1d43144d56b35d62ebf72d1e25" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89"
I20240501 13:59:58.414777 17104 raft_consensus.cc:3032] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:58.414870 17104 raft_consensus.cc:729] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 70ff9aaa9fa2414ea55225aafbd8fd89, State: Running, Role: LEADER
I20240501 13:59:58.415241 17104 consensus_queue.cc:260] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:58.415761 17104 raft_consensus.cc:3037] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:58.419195 17104 raft_consensus.cc:2454] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943a7d1d43144d56b35d62ebf72d1e25 in term 2.
I20240501 13:59:58.424504 18345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9" candidate_uuid: "943a7d1d43144d56b35d62ebf72d1e25" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "200e98827f7c44d29cd71ffd7faec1ca"
I20240501 13:59:58.424762 18345 raft_consensus.cc:3037] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:58.425271 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfe952008f438c887a7d9574fc8bb9" candidate_uuid: "943a7d1d43144d56b35d62ebf72d1e25" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2"
I20240501 13:59:58.425525 18471 raft_consensus.cc:3037] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:58.427034 18345 raft_consensus.cc:2454] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943a7d1d43144d56b35d62ebf72d1e25 in term 2.
I20240501 13:59:58.427570 18149 leader_election.cc:302] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 200e98827f7c44d29cd71ffd7faec1ca, 70ff9aaa9fa2414ea55225aafbd8fd89, 943a7d1d43144d56b35d62ebf72d1e25; no voters:
I20240501 13:59:58.427945 18471 raft_consensus.cc:2454] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943a7d1d43144d56b35d62ebf72d1e25 in term 2.
I20240501 13:59:58.428094 20716 raft_consensus.cc:2781] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:58.428911 20716 raft_consensus.cc:686] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 LEADER]: Becoming Leader. State: Replica: 943a7d1d43144d56b35d62ebf72d1e25, State: Running, Role: LEADER
I20240501 13:59:58.429468 20716 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:58.433559 16572 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 reported cstate change: term changed from 1 to 2, leader changed from 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) to 943a7d1d43144d56b35d62ebf72d1e25 (127.16.36.133). New cstate: current_term: 2 leader_uuid: "943a7d1d43144d56b35d62ebf72d1e25" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:58.693192 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 13:59:58.719827 16778 consensus_peers.cc:485] T dfb457e2b1b14e9688fd392837c0dc92 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 13:59:58.739878 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 13:59:58.816311 18345 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 943a7d1d43144d56b35d62ebf72d1e25: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:58.817013 20716 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 13:59:58.842814 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 13:59:58.846093 17038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 13:59:58.924314 17104 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Refusing update from remote peer 943a7d1d43144d56b35d62ebf72d1e25: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:58.925115 20716 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:58.930096 18471 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Refusing update from remote peer 943a7d1d43144d56b35d62ebf72d1e25: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 13:59:58.932606 20763 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:59.323376 18213 consensus_queue.cc:237] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } }
I20240501 13:59:59.325537 18345 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Refusing update from remote peer 943a7d1d43144d56b35d62ebf72d1e25: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 13:59:59.325567 18471 raft_consensus.cc:1261] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Refusing update from remote peer 943a7d1d43144d56b35d62ebf72d1e25: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 13:59:59.326251 20763 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 13:59:59.326768 20716 consensus_queue.cc:1035] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 13:59:59.329617 20763 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:59.330612 18471 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P efb8dd25e14a46ae89eda6bbe5f940a2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:59.330497 18345 raft_consensus.cc:2932] T a3bfe952008f438c887a7d9574fc8bb9 P 200e98827f7c44d29cd71ffd7faec1ca [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } } }
I20240501 13:59:59.335913 16557 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3bfe952008f438c887a7d9574fc8bb9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:59.337129 16572 catalog_manager.cc:5484] T a3bfe952008f438c887a7d9574fc8bb9 P 943a7d1d43144d56b35d62ebf72d1e25 reported cstate change: config changed from index 11 to 13, VOTER 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132) evicted. New cstate: current_term: 2 leader_uuid: "943a7d1d43144d56b35d62ebf72d1e25" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efb8dd25e14a46ae89eda6bbe5f940a2" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 39399 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200e98827f7c44d29cd71ffd7faec1ca" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 40959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:59.342983 17085 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bfe952008f438c887a7d9574fc8bb9 with delete_type TABLET_DATA_TOMBSTONED (TS 70ff9aaa9fa2414ea55225aafbd8fd89 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:53578
I20240501 13:59:59.343910 20782 tablet_replica.cc:330] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: stopping tablet replica
I20240501 13:59:59.344221 20782 raft_consensus.cc:2229] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:59.344513 20782 raft_consensus.cc:2258] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:59.345993 20782 ts_tablet_manager.cc:1875] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:59.351154 20782 ts_tablet_manager.cc:1888] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 13:59:59.351300 20782 log.cc:1196] T a3bfe952008f438c887a7d9574fc8bb9 P 70ff9aaa9fa2414ea55225aafbd8fd89: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/a3bfe952008f438c887a7d9574fc8bb9
I20240501 13:59:59.352301 16557 catalog_manager.cc:4830] TS 70ff9aaa9fa2414ea55225aafbd8fd89 (127.16.36.132:37337): tablet a3bfe952008f438c887a7d9574fc8bb9 (table rebalance_test_table_1 [id=51e81864112344d586fbc1ce1e10ccab]) successfully deleted
I20240501 13:59:59.594974 18325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.598825 16824 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:59.606125 16954 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:59.607048 18586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.607424 18194 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.637077 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.659574 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 13:59:59.669886 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 13:59:59.884150 18325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.892645 18586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.900403 18194 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:59.921525 16824 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 13:59:59.930088 16954 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 13:59:59.936405 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:59.936401 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:00.057682 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:00:00.227438 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.129:35177
--local_ip_for_outbound_sockets=127.16.36.129
--tserver_master_addrs=127.16.36.190:43309
--webserver_port=44253
--webserver_interface=127.16.36.129
--builtin_ntp_servers=127.16.36.148:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 14:00:00.265728 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:00:00.394017 16908 consensus_peers.cc:485] T 2269a465c771402d882a363a579ba508 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:00:00.468806 16778 consensus_peers.cc:485] T 92cf36a8fccc4cae80751e98374448d8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:00:00.600646 20859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:00.600979 20859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:00.601121 20859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:00.609099 20859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:00:00.609236 20859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:00.609412 20859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.129
I20240501 14:00:00.619014 20859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.129:35177
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.129
--webserver_port=44253
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.190:43309
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.129
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:00.620441 20859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:00.622301 20859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:00.636613 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:00.637076 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:00.637866 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:00.638935 20859 server_base.cc:1034] running on GCE node
I20240501 14:00:00.639664 20859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:00.640780 20859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:00.642055 20859 hybrid_clock.cc:648] HybridClock initialized: now 1714572000641990 us; error 63 us; skew 500 ppm
I20240501 14:00:00.642382 20859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:00.645143 20859 webserver.cc:460] Webserver started at http://127.16.36.129:44253/ using document root <none> and password file <none>
I20240501 14:00:00.645968 20859 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:00.646092 20859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:00.652205 20859 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:00.657390 20876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:00.659031 20859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:00:00.659215 20859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "10f8b1516d5142709756d06b06c5d55f"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-9q9t"
I20240501 14:00:00.660163 20859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:00.688982 20859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:00.689877 20859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:00.690173 20859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:00.691287 20859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:00.693667 20883 ts_tablet_manager.cc:522] Loading tablet metadata (0/11 complete)
I20240501 14:00:00.719287 20859 ts_tablet_manager.cc:559] Loaded tablet metadata (11 total tablets, 11 live tablets)
I20240501 14:00:00.719425 20859 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.026s user 0.002s sys 0.000s
I20240501 14:00:00.719569 20859 ts_tablet_manager.cc:574] Registering tablets (0/11 complete)
I20240501 14:00:00.723708 20883 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:00.740231 20859 ts_tablet_manager.cc:590] Registered 11 tablets
I20240501 14:00:00.740366 20859 ts_tablet_manager.cc:569] Time spent register tablets: real 0.021s user 0.016s sys 0.004s
I20240501 14:00:00.744036 20883 log.cc:824] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:00.797791 20859 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.129:35177
I20240501 14:00:00.797838 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.129:35177 every 8 connection(s)
I20240501 14:00:00.799830 20859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:00:00.809206 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 20859
I20240501 14:00:00.820605 20883 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:00.831559 20883 tablet_bootstrap.cc:492] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:00.845094 20883 ts_tablet_manager.cc:1367] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.122s user 0.083s sys 0.013s
I20240501 14:00:00.854475 20883 raft_consensus.cc:348] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 14:00:00.855684 20883 raft_consensus.cc:729] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:00.856499 20883 consensus_queue.cc:260] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } }
I20240501 14:00:00.868623 20883 ts_tablet_manager.cc:1398] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.023s user 0.015s sys 0.000s
I20240501 14:00:00.869213 20883 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:00.888711 20946 raft_consensus.cc:3037] T 8f27c70cad684568af8d8119559475cc P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:00.900328 20991 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43309
I20240501 14:00:00.900635 20991 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:00.901890 20991 heartbeater.cc:511] Master 127.16.36.190:43309 requested a full tablet report, sending...
W20240501 14:00:00.907006 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 46: this message will repeat every 5th retry.
I20240501 14:00:00.912587 16570 ts_manager.cc:194] Re-registered known tserver with Master: 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177)
I20240501 14:00:00.917131 16570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.129:37141
I20240501 14:00:00.920863 20991 heartbeater.cc:503] Master 127.16.36.190:43309 was elected leader, sending a full tablet report...
I20240501 14:00:00.934190 20883 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:00.934806 20883 tablet_bootstrap.cc:492] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:00.935837 20883 ts_tablet_manager.cc:1367] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.067s user 0.041s sys 0.016s
I20240501 14:00:00.936712 20883 raft_consensus.cc:348] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 14:00:00.936937 20883 raft_consensus.cc:729] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:00.937252 20883 consensus_queue.cc:260] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 14:00:00.937783 20883 ts_tablet_manager.cc:1398] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:00.938112 20883 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:00.978322 20883 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:00.978907 20883 tablet_bootstrap.cc:492] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:00.979970 20883 ts_tablet_manager.cc:1367] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.042s user 0.029s sys 0.011s
I20240501 14:00:00.980684 20883 raft_consensus.cc:348] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 14:00:00.980894 20883 raft_consensus.cc:729] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:00.981142 20883 consensus_queue.cc:260] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 14:00:00.981884 20883 ts_tablet_manager.cc:1398] T 2269a465c771402d882a363a579ba508 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:00:00.982172 20883 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:01.022483 20883 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:01.023257 20883 tablet_bootstrap.cc:492] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.024375 20883 ts_tablet_manager.cc:1367] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.042s user 0.029s sys 0.011s
I20240501 14:00:01.025219 20883 raft_consensus.cc:348] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 14:00:01.025439 20883 raft_consensus.cc:729] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.025712 20883 consensus_queue.cc:260] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 14:00:01.026232 20883 ts_tablet_manager.cc:1398] T dfb457e2b1b14e9688fd392837c0dc92 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:01.026546 20883 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:01.081578 20883 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240501 14:00:01.082055 16778 consensus_peers.cc:485] T f9c8c82da2da4173899f5b9e03894da8 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 46: this message will repeat every 5th retry.
I20240501 14:00:01.082454 20883 tablet_bootstrap.cc:492] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.083869 20883 ts_tablet_manager.cc:1367] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.057s user 0.042s sys 0.008s
I20240501 14:00:01.084561 20883 raft_consensus.cc:348] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 14:00:01.084846 20883 raft_consensus.cc:729] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.085088 20883 consensus_queue.cc:260] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 14:00:01.085798 20883 ts_tablet_manager.cc:1398] T 92cf36a8fccc4cae80751e98374448d8 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:01.086081 20883 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:01.132932 20883 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:01.133653 20883 tablet_bootstrap.cc:492] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.134979 20883 ts_tablet_manager.cc:1367] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.049s user 0.037s sys 0.011s
I20240501 14:00:01.135792 20883 raft_consensus.cc:348] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.135995 20883 raft_consensus.cc:729] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.136309 20883 consensus_queue.cc:260] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.136900 20883 ts_tablet_manager.cc:1398] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:01.137187 20883 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:01.178350 20883 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:01.179028 20883 tablet_bootstrap.cc:492] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.180209 20883 ts_tablet_manager.cc:1367] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.043s user 0.033s sys 0.007s
I20240501 14:00:01.181207 20883 raft_consensus.cc:348] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.181488 20883 raft_consensus.cc:729] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.181788 20883 consensus_queue.cc:260] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.182569 20883 ts_tablet_manager.cc:1398] T ac5a6954888149209b7b2fed2ab849f2 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:01.183104 20883 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:01.227206 20883 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:01.227792 20883 tablet_bootstrap.cc:492] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.228816 20883 ts_tablet_manager.cc:1367] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.046s user 0.032s sys 0.012s
I20240501 14:00:01.229550 20883 raft_consensus.cc:348] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.229730 20883 raft_consensus.cc:729] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.230006 20883 consensus_queue.cc:260] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.230521 20883 ts_tablet_manager.cc:1398] T a25c0fb5b5264854a229ed8acdbed791 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:01.230845 20883 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:01.271579 20883 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:01.272143 20883 tablet_bootstrap.cc:492] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.273125 20883 ts_tablet_manager.cc:1367] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.042s user 0.035s sys 0.004s
I20240501 14:00:01.273820 20883 raft_consensus.cc:348] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 14:00:01.274042 20883 raft_consensus.cc:729] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.274282 20883 consensus_queue.cc:260] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } }
I20240501 14:00:01.274878 20883 ts_tablet_manager.cc:1398] T 35a493549e9a4975a3820fabb6e6e491 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:01.275313 20883 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
W20240501 14:00:01.305498 16778 consensus_peers.cc:485] T 55f551f8187a49a4afa2e64836a12451 P d871186c2b234914ab63dc5e136f0f03 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 46: this message will repeat every 5th retry.
I20240501 14:00:01.326678 20946 raft_consensus.cc:3037] T 3068f422a4d54d998715bffa2667854e P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:01.340754 20883 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:01.341545 20883 tablet_bootstrap.cc:492] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.342861 20883 ts_tablet_manager.cc:1367] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.068s user 0.043s sys 0.023s
I20240501 14:00:01.343943 20883 raft_consensus.cc:348] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 14:00:01.344188 20883 raft_consensus.cc:729] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.344466 20883 consensus_queue.cc:260] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } } peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } }
I20240501 14:00:01.345047 20883 ts_tablet_manager.cc:1398] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:01.345345 20883 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap starting.
I20240501 14:00:01.409646 20883 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:01.410377 20883 tablet_bootstrap.cc:492] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Bootstrap complete.
I20240501 14:00:01.411785 20883 ts_tablet_manager.cc:1367] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Time spent bootstrapping tablet: real 0.066s user 0.053s sys 0.011s
I20240501 14:00:01.412739 20883 raft_consensus.cc:348] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.413024 20883 raft_consensus.cc:729] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f8b1516d5142709756d06b06c5d55f, State: Initialized, Role: FOLLOWER
I20240501 14:00:01.413329 20883 consensus_queue.cc:260] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70ff9aaa9fa2414ea55225aafbd8fd89" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 37337 } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } }
I20240501 14:00:01.413978 20883 ts_tablet_manager.cc:1398] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:01.516155 18325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:01.538074 20926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:01.556319 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.561467 16824 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:01.571624 20946 raft_consensus.cc:3037] T f9c8c82da2da4173899f5b9e03894da8 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:01.574980 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:01.582013 18194 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:01.591603 18586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:01.655882 16954 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:01.758939 20946 raft_consensus.cc:3037] T 55f551f8187a49a4afa2e64836a12451 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:01.931499 20926 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:01.943257 18586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:01.953222 18325 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:01.956328 17085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:01.964522 18194 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:01.980683 16824 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:01.986749 16954 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:02.026010 18455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:02.158448 20999 raft_consensus.cc:482] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:02.158767 20999 raft_consensus.cc:504] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f4b54fe4701746c0930c9b0667c90519" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 37163 } } peers { permanent_uuid: "10f8b1516d5142709756d06b06c5d55f" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 35177 } attrs { replace: true } } peers { permanent_uuid: "d871186c2b234914ab63dc5e136f0f03" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 42745 } } peers { permanent_uuid: "943a7d1d43144d56b35d62ebf72d1e25" member_type: NON_VOTER last_known_addr { host: "127.16.36.133" port: 36049 } attrs { promote: true } }
I20240501 14:00:02.160553 20999 leader_election.cc:288] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f4b54fe4701746c0930c9b0667c90519 (127.16.36.131:37163), d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745)
I20240501 14:00:02.169795 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f4b54fe4701746c0930c9b0667c90519" is_pre_election: true
I20240501 14:00:02.170091 16974 raft_consensus.cc:1789] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.10
I20240501 14:00:02.170279 16974 raft_consensus.cc:1799] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Handling vote request from an unknown peer 10f8b1516d5142709756d06b06c5d55f
I20240501 14:00:02.170492 16974 raft_consensus.cc:2379] T 101e3579d2954f2aac17d0f10ac84865 P f4b54fe4701746c0930c9b0667c90519 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10f8b1516d5142709756d06b06c5d55f in current term 2: Already voted for candidate d871186c2b234914ab63dc5e136f0f03 in this term.
I20240501 14:00:02.170876 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101e3579d2954f2aac17d0f10ac84865" candidate_uuid: "10f8b1516d5142709756d06b06c5d55f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d871186c2b234914ab63dc5e136f0f03" is_pre_election: true
I20240501 14:00:02.171656 20880 leader_election.cc:302] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f8b1516d5142709756d06b06c5d55f; no voters: d871186c2b234914ab63dc5e136f0f03, f4b54fe4701746c0930c9b0667c90519
I20240501 14:00:02.172216 20999 raft_consensus.cc:3037] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:02.174610 20999 raft_consensus.cc:2726] T 101e3579d2954f2aac17d0f10ac84865 P 10f8b1516d5142709756d06b06c5d55f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:00:02.217352 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 20859
I20240501 14:00:02.224848 20985 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:02.342886 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 20859
W20240501 14:00:02.368358 16908 connection.cc:535] client connection to 127.16.36.129:35177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:00:02.368561 16908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 27 similar messages]
I20240501 14:00:02.368881 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 16763
W20240501 14:00:02.370826 16908 consensus_peers.cc:485] T ac5a6954888149209b7b2fed2ab849f2 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:00:02.370958 16908 consensus_peers.cc:485] T 35a493549e9a4975a3820fabb6e6e491 P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:02.375267 16883 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:00:02.529703 16908 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:00:02.562669 16778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20240501 14:00:02.565932 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 16763
I20240501 14:00:02.598834 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 16893
W20240501 14:00:02.599499 16911 consensus_peers.cc:485] T 3068f422a4d54d998715bffa2667854e P f4b54fe4701746c0930c9b0667c90519 -> Peer d871186c2b234914ab63dc5e136f0f03 (127.16.36.130:42745): Couldn't send request to peer d871186c2b234914ab63dc5e136f0f03. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.130:42745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:02.605199 17014 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:02.794162 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 16893
W20240501 14:00:02.816468 17038 consensus_peers.cc:485] T a25c0fb5b5264854a229ed8acdbed791 P 70ff9aaa9fa2414ea55225aafbd8fd89 -> Peer 10f8b1516d5142709756d06b06c5d55f (127.16.36.129:35177): Couldn't send request to peer 10f8b1516d5142709756d06b06c5d55f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:35177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:02.826723 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 17023
I20240501 14:00:02.833683 17145 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.009127 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 17023
I20240501 14:00:03.039182 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 18131
I20240501 14:00:03.045985 18252 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.189065 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 18131
W20240501 14:00:03.213230 18281 connection.cc:535] server connection from 127.16.36.133:43999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:00:03.213687 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 18263
I20240501 14:00:03.220657 18385 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.362565 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 18263
W20240501 14:00:03.386860 16557 connection.cc:535] server connection from 127.16.36.134:44199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:00:03.387287 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 18393
I20240501 14:00:03.394208 18513 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.533975 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 18393
W20240501 14:00:03.541657 18541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:40959: connect: Connection refused (error 111)
W20240501 14:00:03.543431 18541 consensus_peers.cc:485] T 5f408fdebfb84082b0a55ad059e90694 P 2e5565b8761548539de35f9318bec337 -> Peer 200e98827f7c44d29cd71ffd7faec1ca (127.16.36.134:40959): Couldn't send request to peer 200e98827f7c44d29cd71ffd7faec1ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:40959: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:03.560441 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 18523
I20240501 14:00:03.567337 18643 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.710530 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 18523
I20240501 14:00:03.736078 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 16542
I20240501 14:00:03.737432 16602 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:03.923148 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 16542
2024-05-01T14:00:03Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (50756 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (50756 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240501 14:00:03.997632 16530 test_util.cc:268] Using random seed: -415402088
I20240501 14:00:04.000469 16530 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:00:04.000703 16530 ts_itest-base.cc:113] --------------
I20240501 14:00:04.000764 16530 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:00:04.000862 16530 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:00:04.000921 16530 ts_itest-base.cc:116] --------------
2024-05-01T14:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:00:04Z Disabled control of system clock
I20240501 14:00:04.039032 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:42127
--webserver_interface=127.16.36.190
--webserver_port=0
--builtin_ntp_servers=127.16.36.148:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.190:42127
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:00:04.398000 21147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:04.398351 21147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:04.398500 21147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:04.406414 21147 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:04.406566 21147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:00:04.406620 21147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:04.406733 21147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:00:04.406792 21147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:00:04.416798 21147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.190:42127
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:04.418279 21147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:04.420473 21147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:04.432693 21152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:04.433290 21153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:04.434053 21155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:04.435081 21147 server_base.cc:1034] running on GCE node
I20240501 14:00:04.435989 21147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:04.437629 21147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:04.438858 21147 hybrid_clock.cc:648] HybridClock initialized: now 1714572004438806 us; error 47 us; skew 500 ppm
I20240501 14:00:04.439245 21147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:04.445542 21147 webserver.cc:460] Webserver started at http://127.16.36.190:33389/ using document root <none> and password file <none>
I20240501 14:00:04.446432 21147 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:04.446565 21147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:04.446947 21147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:04.449910 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2ea4d848178c4d6693f956c1ce00ff51"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-9q9t"
I20240501 14:00:04.450969 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2ea4d848178c4d6693f956c1ce00ff51"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-9q9t"
I20240501 14:00:04.456859 21147 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240501 14:00:04.460572 21161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:04.461939 21147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:04.462136 21147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ea4d848178c4d6693f956c1ce00ff51"
format_stamp: "Formatted at 2024-05-01 14:00:04 on dist-test-slave-9q9t"
I20240501 14:00:04.462366 21147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:04.488477 21147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:04.489495 21147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:04.489804 21147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:04.509966 21147 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.190:42127
I20240501 14:00:04.510018 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.190:42127 every 8 connection(s)
I20240501 14:00:04.511559 21147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:00:04.515794 21213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.522044 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 21147
I20240501 14:00:04.522358 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:00:04.527830 21213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51: Bootstrap starting.
I20240501 14:00:04.531986 21213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:04.533492 21213 log.cc:824] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:04.536927 21213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51: No bootstrap required, opened a new log
I20240501 14:00:04.543937 21213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea4d848178c4d6693f956c1ce00ff51" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 42127 } }
I20240501 14:00:04.544425 21213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:04.544549 21213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea4d848178c4d6693f956c1ce00ff51, State: Initialized, Role: FOLLOWER
I20240501 14:00:04.545498 21213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea4d848178c4d6693f956c1ce00ff51" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 42127 } }
I20240501 14:00:04.545781 21213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:00:04.545928 21213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:00:04.546103 21213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.548674 21213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea4d848178c4d6693f956c1ce00ff51" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 42127 } }
I20240501 14:00:04.549253 21213 leader_election.cc:302] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ea4d848178c4d6693f956c1ce00ff51; no voters:
I20240501 14:00:04.549942 21213 leader_election.cc:288] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:00:04.550132 21218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:04.550884 21218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [term 1 LEADER]: Becoming Leader. State: Replica: 2ea4d848178c4d6693f956c1ce00ff51, State: Running, Role: LEADER
I20240501 14:00:04.551602 21218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea4d848178c4d6693f956c1ce00ff51" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 42127 } }
I20240501 14:00:04.552407 21213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:00:04.555219 21220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ea4d848178c4d6693f956c1ce00ff51. Latest consensus state: current_term: 1 leader_uuid: "2ea4d848178c4d6693f956c1ce00ff51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea4d848178c4d6693f956c1ce00ff51" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 42127 } } }
I20240501 14:00:04.555189 21219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ea4d848178c4d6693f956c1ce00ff51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea4d848178c4d6693f956c1ce00ff51" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 42127 } } }
I20240501 14:00:04.555910 21219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:04.560070 21220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:04.561290 21227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:00:04.566053 21227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:00:04.576190 21227 catalog_manager.cc:1347] Generated new cluster ID: e2e04316c14f4665ad1d95c8aa9f195f
I20240501 14:00:04.576364 21227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:00:04.590250 21227 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:00:04.591675 21227 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:00:04.600494 21227 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2ea4d848178c4d6693f956c1ce00ff51: Generated new TSK 0
I20240501 14:00:04.601572 21227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:00:04.617792 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.129:0
--local_ip_for_outbound_sockets=127.16.36.129
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--builtin_ntp_servers=127.16.36.148:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:00:04.973712 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:04.974043 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:04.974136 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:04.981902 21237 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:04.982062 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:04.982193 21237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.129
I20240501 14:00:04.991914 21237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.129
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:04.993346 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:04.995541 21237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:05.009857 21242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.010660 21245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.011507 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:05.012277 21237 server_base.cc:1034] running on GCE node
I20240501 14:00:05.012890 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:05.013886 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:05.015090 21237 hybrid_clock.cc:648] HybridClock initialized: now 1714572005015033 us; error 48 us; skew 500 ppm
I20240501 14:00:05.015401 21237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:05.017814 21237 webserver.cc:460] Webserver started at http://127.16.36.129:36923/ using document root <none> and password file <none>
I20240501 14:00:05.018576 21237 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:05.018718 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:05.019045 21237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:05.021737 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fac9d709ecfe49ac8bc8b34c2e22568d"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:05.022564 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fac9d709ecfe49ac8bc8b34c2e22568d"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:05.028400 21237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 14:00:05.031998 21251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.033450 21237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:05.033634 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fac9d709ecfe49ac8bc8b34c2e22568d"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:05.033849 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:05.060107 21237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:05.061053 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:05.061348 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:05.062389 21237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:05.063993 21237 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:05.064103 21237 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.064227 21237 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:05.064307 21237 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.097380 21237 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.129:43259
I20240501 14:00:05.097446 21363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.129:43259 every 8 connection(s)
I20240501 14:00:05.098923 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:00:05.108644 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 21237
I20240501 14:00:05.108845 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:00:05.112989 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.130:0
--local_ip_for_outbound_sockets=127.16.36.130
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--builtin_ntp_servers=127.16.36.148:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:05.113219 21364 heartbeater.cc:347] Connected to a master server at 127.16.36.190:42127
I20240501 14:00:05.113584 21364 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:05.114449 21364 heartbeater.cc:511] Master 127.16.36.190:42127 requested a full tablet report, sending...
I20240501 14:00:05.117403 21178 ts_manager.cc:194] Registered new tserver with Master: fac9d709ecfe49ac8bc8b34c2e22568d (127.16.36.129:43259)
I20240501 14:00:05.119285 21178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.129:53905
W20240501 14:00:05.464085 21368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:05.464404 21368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:05.464557 21368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:05.472575 21368 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:05.472718 21368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:05.472884 21368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.130
I20240501 14:00:05.482307 21368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.130
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:05.483753 21368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:05.485616 21368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:05.499085 21374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.499617 21376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.500077 21373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:05.501097 21368 server_base.cc:1034] running on GCE node
I20240501 14:00:05.501796 21368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:05.502904 21368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:05.504154 21368 hybrid_clock.cc:648] HybridClock initialized: now 1714572005504072 us; error 65 us; skew 500 ppm
I20240501 14:00:05.504539 21368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:05.507238 21368 webserver.cc:460] Webserver started at http://127.16.36.130:42921/ using document root <none> and password file <none>
I20240501 14:00:05.508062 21368 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:05.508167 21368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:05.508518 21368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:05.512099 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fac8fbd2359e412babe5a8a8c60dec1a"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:05.513202 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fac8fbd2359e412babe5a8a8c60dec1a"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:05.520359 21368 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240501 14:00:05.525306 21382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.527246 21368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:05.527462 21368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fac8fbd2359e412babe5a8a8c60dec1a"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:05.527701 21368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:05.551131 21368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:05.552052 21368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:05.552316 21368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:05.553570 21368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:05.555276 21368 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:05.555397 21368 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.555487 21368 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:05.555543 21368 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:05.589931 21368 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.130:32819
I20240501 14:00:05.589999 21494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.130:32819 every 8 connection(s)
I20240501 14:00:05.591399 21368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:00:05.596320 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 21368
I20240501 14:00:05.596508 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:00:05.600185 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.131:0
--local_ip_for_outbound_sockets=127.16.36.131
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--builtin_ntp_servers=127.16.36.148:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:05.608353 21495 heartbeater.cc:347] Connected to a master server at 127.16.36.190:42127
I20240501 14:00:05.608703 21495 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:05.609551 21495 heartbeater.cc:511] Master 127.16.36.190:42127 requested a full tablet report, sending...
I20240501 14:00:05.611135 21178 ts_manager.cc:194] Registered new tserver with Master: fac8fbd2359e412babe5a8a8c60dec1a (127.16.36.130:32819)
I20240501 14:00:05.611943 21178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.130:54301
W20240501 14:00:05.949579 21498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:05.949927 21498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:05.950090 21498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:05.958185 21498 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:05.958355 21498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:05.958526 21498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.131
I20240501 14:00:05.968276 21498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.131
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:05.969731 21498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:05.971735 21498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:05.985467 21503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.986212 21504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:05.986908 21506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:05.987622 21498 server_base.cc:1034] running on GCE node
I20240501 14:00:05.988272 21498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:05.989262 21498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:05.990520 21498 hybrid_clock.cc:648] HybridClock initialized: now 1714572005990463 us; error 50 us; skew 500 ppm
I20240501 14:00:05.990907 21498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:05.993427 21498 webserver.cc:460] Webserver started at http://127.16.36.131:45377/ using document root <none> and password file <none>
I20240501 14:00:05.994235 21498 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:05.994362 21498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:05.994752 21498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:05.997611 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "441f63d4cdde41b5ba67c3d4c0d7277b"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:05.998567 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "441f63d4cdde41b5ba67c3d4c0d7277b"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:06.004431 21498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240501 14:00:06.008193 21512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.009711 21498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240501 14:00:06.009889 21498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "441f63d4cdde41b5ba67c3d4c0d7277b"
format_stamp: "Formatted at 2024-05-01 14:00:05 on dist-test-slave-9q9t"
I20240501 14:00:06.010113 21498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:06.035185 21498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:06.036096 21498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:06.036383 21498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:06.037420 21498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:06.039103 21498 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:06.039211 21498 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.039338 21498 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:06.039425 21498 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.071521 21498 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.131:39133
I20240501 14:00:06.071579 21624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.131:39133 every 8 connection(s)
I20240501 14:00:06.073042 21498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:00:06.082536 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 21498
I20240501 14:00:06.082751 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:00:06.086439 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:0
--local_ip_for_outbound_sockets=127.16.36.132
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--builtin_ntp_servers=127.16.36.148:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:06.087020 21625 heartbeater.cc:347] Connected to a master server at 127.16.36.190:42127
I20240501 14:00:06.087352 21625 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:06.088225 21625 heartbeater.cc:511] Master 127.16.36.190:42127 requested a full tablet report, sending...
I20240501 14:00:06.089931 21178 ts_manager.cc:194] Registered new tserver with Master: 441f63d4cdde41b5ba67c3d4c0d7277b (127.16.36.131:39133)
I20240501 14:00:06.091125 21178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.131:59631
I20240501 14:00:06.123314 21364 heartbeater.cc:503] Master 127.16.36.190:42127 was elected leader, sending a full tablet report...
W20240501 14:00:06.439596 21628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:06.439941 21628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:06.440150 21628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:06.447791 21628 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:06.447989 21628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:06.448190 21628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 14:00:06.457866 21628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:06.459313 21628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:06.461112 21628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:06.474081 21633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.474901 21636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.475675 21634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:06.476457 21628 server_base.cc:1034] running on GCE node
I20240501 14:00:06.477169 21628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:06.478226 21628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:06.479480 21628 hybrid_clock.cc:648] HybridClock initialized: now 1714572006479405 us; error 57 us; skew 500 ppm
I20240501 14:00:06.479871 21628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:06.482470 21628 webserver.cc:460] Webserver started at http://127.16.36.132:37889/ using document root <none> and password file <none>
I20240501 14:00:06.483368 21628 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:06.483495 21628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:06.483877 21628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:06.486855 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2c0b95fa1214469c9824752e1438626f"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-9q9t"
I20240501 14:00:06.487766 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2c0b95fa1214469c9824752e1438626f"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-9q9t"
I20240501 14:00:06.494035 21628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240501 14:00:06.498080 21642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.499679 21628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:06.499858 21628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2c0b95fa1214469c9824752e1438626f"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-9q9t"
I20240501 14:00:06.500089 21628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:06.530858 21628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:06.531780 21628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:06.532066 21628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:06.533093 21628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:06.534751 21628 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:06.534854 21628 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.534970 21628 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:06.535043 21628 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.574136 21628 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:41725
I20240501 14:00:06.574208 21754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:41725 every 8 connection(s)
I20240501 14:00:06.575595 21628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:00:06.577839 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 21628
I20240501 14:00:06.578048 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:00:06.581612 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:0
--local_ip_for_outbound_sockets=127.16.36.133
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--builtin_ntp_servers=127.16.36.148:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:06.592165 21755 heartbeater.cc:347] Connected to a master server at 127.16.36.190:42127
I20240501 14:00:06.592559 21755 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:06.593580 21755 heartbeater.cc:511] Master 127.16.36.190:42127 requested a full tablet report, sending...
I20240501 14:00:06.595335 21178 ts_manager.cc:194] Registered new tserver with Master: 2c0b95fa1214469c9824752e1438626f (127.16.36.132:41725)
I20240501 14:00:06.596417 21178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:44595
I20240501 14:00:06.614827 21495 heartbeater.cc:503] Master 127.16.36.190:42127 was elected leader, sending a full tablet report...
W20240501 14:00:06.923081 21758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:06.923429 21758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:06.923580 21758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:06.931365 21758 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:06.931538 21758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:06.931706 21758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 14:00:06.941362 21758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:42127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:06.942800 21758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:06.944574 21758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:06.957186 21763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.957837 21764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:06.958768 21766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:06.959668 21758 server_base.cc:1034] running on GCE node
I20240501 14:00:06.960386 21758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:06.961419 21758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:06.962718 21758 hybrid_clock.cc:648] HybridClock initialized: now 1714572006962659 us; error 43 us; skew 500 ppm
I20240501 14:00:06.963094 21758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:06.965664 21758 webserver.cc:460] Webserver started at http://127.16.36.133:39881/ using document root <none> and password file <none>
I20240501 14:00:06.966593 21758 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:06.966778 21758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:06.967175 21758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:06.970266 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c6792b99683f496ba2d80dbe7ed5bcbf"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-9q9t"
I20240501 14:00:06.971262 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c6792b99683f496ba2d80dbe7ed5bcbf"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-9q9t"
I20240501 14:00:06.977509 21758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240501 14:00:06.981333 21772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:06.982968 21758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:06.983156 21758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c6792b99683f496ba2d80dbe7ed5bcbf"
format_stamp: "Formatted at 2024-05-01 14:00:06 on dist-test-slave-9q9t"
I20240501 14:00:06.983377 21758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:07.003602 21758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:07.004648 21758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:07.004940 21758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:07.006052 21758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:07.007922 21758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:07.008034 21758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.008162 21758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:07.008237 21758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:07.040731 21758 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:38113
I20240501 14:00:07.040787 21884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:38113 every 8 connection(s)
I20240501 14:00:07.042076 21758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:00:07.043304 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 21758
I20240501 14:00:07.043486 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:00:07.055799 21885 heartbeater.cc:347] Connected to a master server at 127.16.36.190:42127
I20240501 14:00:07.056154 21885 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:07.057005 21885 heartbeater.cc:511] Master 127.16.36.190:42127 requested a full tablet report, sending...
I20240501 14:00:07.058511 21178 ts_manager.cc:194] Registered new tserver with Master: c6792b99683f496ba2d80dbe7ed5bcbf (127.16.36.133:38113)
I20240501 14:00:07.059401 21178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:39619
I20240501 14:00:07.060170 16530 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:00:07.078090 21177 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45716:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:00:07.094048 21625 heartbeater.cc:503] Master 127.16.36.190:42127 was elected leader, sending a full tablet report...
I20240501 14:00:07.137900 21690 tablet_service.cc:1467] Processing CreateTablet for tablet ceb625b77f3a4d1ea1b341981abb8394 (DEFAULT_TABLE table=TestTable [id=219eaa46540640148f8863a553edfc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.139672 21690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb625b77f3a4d1ea1b341981abb8394. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.145603 21820 tablet_service.cc:1467] Processing CreateTablet for tablet ceb625b77f3a4d1ea1b341981abb8394 (DEFAULT_TABLE table=TestTable [id=219eaa46540640148f8863a553edfc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.146018 21299 tablet_service.cc:1467] Processing CreateTablet for tablet ceb625b77f3a4d1ea1b341981abb8394 (DEFAULT_TABLE table=TestTable [id=219eaa46540640148f8863a553edfc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.147464 21299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb625b77f3a4d1ea1b341981abb8394. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.147491 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb625b77f3a4d1ea1b341981abb8394. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.154215 21903 tablet_bootstrap.cc:492] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f: Bootstrap starting.
I20240501 14:00:07.158469 21903 tablet_bootstrap.cc:654] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.159549 21903 log.cc:824] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:07.161792 21905 tablet_bootstrap.cc:492] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf: Bootstrap starting.
I20240501 14:00:07.162492 21903 tablet_bootstrap.cc:492] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f: No bootstrap required, opened a new log
I20240501 14:00:07.162726 21906 tablet_bootstrap.cc:492] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d: Bootstrap starting.
I20240501 14:00:07.162967 21903 ts_tablet_manager.cc:1367] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 14:00:07.166199 21905 tablet_bootstrap.cc:654] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.166785 21906 tablet_bootstrap.cc:654] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.167773 21905 log.cc:824] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:07.168105 21906 log.cc:824] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:07.170703 21906 tablet_bootstrap.cc:492] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d: No bootstrap required, opened a new log
I20240501 14:00:07.170887 21905 tablet_bootstrap.cc:492] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf: No bootstrap required, opened a new log
I20240501 14:00:07.171005 21906 ts_tablet_manager.cc:1367] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 14:00:07.171182 21905 ts_tablet_manager.cc:1367] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240501 14:00:07.172482 21903 raft_consensus.cc:348] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.172967 21903 raft_consensus.cc:374] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.173130 21903 raft_consensus.cc:729] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0b95fa1214469c9824752e1438626f, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.173936 21903 consensus_queue.cc:260] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.175205 21755 heartbeater.cc:503] Master 127.16.36.190:42127 was elected leader, sending a full tablet report...
I20240501 14:00:07.175864 21903 ts_tablet_manager.cc:1398] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20240501 14:00:07.178225 21906 raft_consensus.cc:348] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.178778 21906 raft_consensus.cc:374] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.178901 21906 raft_consensus.cc:729] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac9d709ecfe49ac8bc8b34c2e22568d, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.180011 21906 consensus_queue.cc:260] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.180440 21905 raft_consensus.cc:348] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.180963 21905 raft_consensus.cc:374] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.181071 21905 raft_consensus.cc:729] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6792b99683f496ba2d80dbe7ed5bcbf, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.181845 21905 consensus_queue.cc:260] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.182921 21906 ts_tablet_manager.cc:1398] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20240501 14:00:07.187425 21885 heartbeater.cc:503] Master 127.16.36.190:42127 was elected leader, sending a full tablet report...
I20240501 14:00:07.188299 21905 ts_tablet_manager.cc:1398] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf: Time spent starting tablet: real 0.017s user 0.006s sys 0.008s
I20240501 14:00:07.271816 21910 raft_consensus.cc:482] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:07.272092 21910 raft_consensus.cc:504] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.273910 21910 leader_election.cc:288] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c0b95fa1214469c9824752e1438626f (127.16.36.132:41725), c6792b99683f496ba2d80dbe7ed5bcbf (127.16.36.133:38113)
I20240501 14:00:07.281339 21710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb625b77f3a4d1ea1b341981abb8394" candidate_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0b95fa1214469c9824752e1438626f" is_pre_election: true
I20240501 14:00:07.281342 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb625b77f3a4d1ea1b341981abb8394" candidate_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" is_pre_election: true
I20240501 14:00:07.281863 21710 raft_consensus.cc:2454] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac9d709ecfe49ac8bc8b34c2e22568d in term 0.
I20240501 14:00:07.281858 21840 raft_consensus.cc:2454] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac9d709ecfe49ac8bc8b34c2e22568d in term 0.
I20240501 14:00:07.282539 21254 leader_election.cc:302] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6792b99683f496ba2d80dbe7ed5bcbf, fac9d709ecfe49ac8bc8b34c2e22568d; no voters:
I20240501 14:00:07.282994 21910 raft_consensus.cc:2781] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:07.283167 21910 raft_consensus.cc:482] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:07.283278 21910 raft_consensus.cc:3037] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.285444 21910 raft_consensus.cc:504] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
I20240501 14:00:07.286087 21910 leader_election.cc:288] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [CANDIDATE]: Term 1 election: Requested vote from peers 2c0b95fa1214469c9824752e1438626f (127.16.36.132:41725), c6792b99683f496ba2d80dbe7ed5bcbf (127.16.36.133:38113)
I20240501 14:00:07.286712 21710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb625b77f3a4d1ea1b341981abb8394" candidate_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0b95fa1214469c9824752e1438626f"
I20240501 14:00:07.286733 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb625b77f3a4d1ea1b341981abb8394" candidate_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf"
I20240501 14:00:07.286931 21840 raft_consensus.cc:3037] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.286957 21710 raft_consensus.cc:3037] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.289359 21840 raft_consensus.cc:2454] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac9d709ecfe49ac8bc8b34c2e22568d in term 1.
I20240501 14:00:07.289446 21710 raft_consensus.cc:2454] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac9d709ecfe49ac8bc8b34c2e22568d in term 1.
I20240501 14:00:07.289884 21254 leader_election.cc:302] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6792b99683f496ba2d80dbe7ed5bcbf, fac9d709ecfe49ac8bc8b34c2e22568d; no voters:
I20240501 14:00:07.290334 21910 raft_consensus.cc:2781] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:07.290815 21910 raft_consensus.cc:686] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [term 1 LEADER]: Becoming Leader. State: Replica: fac9d709ecfe49ac8bc8b34c2e22568d, State: Running, Role: LEADER
I20240501 14:00:07.291380 21910 consensus_queue.cc:237] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } }
W20240501 14:00:07.294049 21886 tablet.cc:2369] T ceb625b77f3a4d1ea1b341981abb8394 P c6792b99683f496ba2d80dbe7ed5bcbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:07.296267 21177 catalog_manager.cc:5484] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d reported cstate change: term changed from 0 to 1, leader changed from <none> to fac9d709ecfe49ac8bc8b34c2e22568d (127.16.36.129). New cstate: current_term: 1 leader_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 } } peers { permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 } } peers { permanent_uuid: "fac9d709ecfe49ac8bc8b34c2e22568d" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 43259 } } }
I20240501 14:00:07.316478 16530 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:00:07.319713 16530 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fac9d709ecfe49ac8bc8b34c2e22568d to finish bootstrapping
W20240501 14:00:07.327668 21756 tablet.cc:2369] T ceb625b77f3a4d1ea1b341981abb8394 P 2c0b95fa1214469c9824752e1438626f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:07.329807 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fac8fbd2359e412babe5a8a8c60dec1a to finish bootstrapping
I20240501 14:00:07.339897 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 441f63d4cdde41b5ba67c3d4c0d7277b to finish bootstrapping
I20240501 14:00:07.350040 16530 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2c0b95fa1214469c9824752e1438626f to finish bootstrapping
W20240501 14:00:07.352593 21365 tablet.cc:2369] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:07.358744 16530 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c6792b99683f496ba2d80dbe7ed5bcbf to finish bootstrapping
I20240501 14:00:07.845539 21946 consensus_queue.cc:1035] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6792b99683f496ba2d80dbe7ed5bcbf" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:07.849906 21560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:07.862843 21950 consensus_queue.cc:1035] T ceb625b77f3a4d1ea1b341981abb8394 P fac9d709ecfe49ac8bc8b34c2e22568d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0b95fa1214469c9824752e1438626f" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:07.869818 21690 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:07.875763 21430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:07.902057 21299 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:07.902141 21820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.063647 21690 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.072269 21560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.085409 21820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.103925 21430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.134804 21299 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:08.743775 21560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.749285 21820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.752946 21690 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.757645 21299 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:08.764262 21430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.924530 21560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.927274 21690 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.935784 21430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.937855 21820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:08.953599 21299 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:09.120497 21430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:09.130650 21690 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:09.130784 21560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:09.132239 21820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:09.134109 21299 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:09.297255 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 21237
I20240501 14:00:09.298678 21359 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:09.398216 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 21237
W20240501 14:00:09.417263 21773 connection.cc:535] server connection from 127.16.36.129:38021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:00:09.417601 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 21368
W20240501 14:00:09.417606 21163 connection.cc:535] server connection from 127.16.36.129:53905 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:00:09.418706 21489 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:09.511992 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 21368
I20240501 14:00:09.530894 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 21498
I20240501 14:00:09.532022 21619 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:09.624925 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 21498
I20240501 14:00:09.643314 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 21628
I20240501 14:00:09.644403 21749 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:09.741605 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 21628
I20240501 14:00:09.759788 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 21758
I20240501 14:00:09.760931 21879 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:09.858434 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 21758
I20240501 14:00:09.876860 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 21147
I20240501 14:00:09.877967 21208 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:10.006919 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 21147
2024-05-01T14:00:10Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (6054 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6054 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240501 14:00:10.052276 16530 test_util.cc:268] Using random seed: -409347435
I20240501 14:00:10.054293 16530 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:00:10.054370 16530 ts_itest-base.cc:113] --------------
I20240501 14:00:10.054445 16530 ts_itest-base.cc:114] 7 tablet servers
I20240501 14:00:10.054495 16530 ts_itest-base.cc:115] 2 replicas per TS
I20240501 14:00:10.054689 16530 ts_itest-base.cc:116] --------------
2024-05-01T14:00:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:00:10Z Disabled control of system clock
I20240501 14:00:10.080569 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:35677
--webserver_interface=127.16.36.190
--webserver_port=0
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.190:35677
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 14:00:10.426226 22110 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:00:10.426654 22110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:10.426797 22110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:10.426921 22110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:10.435204 22110 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:10.435360 22110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:00:10.435482 22110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:10.435551 22110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:00:10.435642 22110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:00:10.445109 22110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.190:35677
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:35677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:10.446563 22110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:10.448524 22110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:10.460213 22115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:10.461062 22118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:10.461758 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:10.462611 22110 server_base.cc:1034] running on GCE node
I20240501 14:00:10.463555 22110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:10.465092 22110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:10.466358 22110 hybrid_clock.cc:648] HybridClock initialized: now 1714572010466271 us; error 66 us; skew 500 ppm
I20240501 14:00:10.466795 22110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:10.473599 22110 webserver.cc:460] Webserver started at http://127.16.36.190:46285/ using document root <none> and password file <none>
I20240501 14:00:10.474457 22110 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:10.474608 22110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:10.475010 22110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:10.477931 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e95d660c7bd7435fa08604fea52664ad"
format_stamp: "Formatted at 2024-05-01 14:00:10 on dist-test-slave-9q9t"
I20240501 14:00:10.478829 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e95d660c7bd7435fa08604fea52664ad"
format_stamp: "Formatted at 2024-05-01 14:00:10 on dist-test-slave-9q9t"
I20240501 14:00:10.484611 22110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20240501 14:00:10.488265 22124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:10.489674 22110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:10.489854 22110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e95d660c7bd7435fa08604fea52664ad"
format_stamp: "Formatted at 2024-05-01 14:00:10 on dist-test-slave-9q9t"
I20240501 14:00:10.490083 22110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:10.513823 22110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:10.514796 22110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:10.515110 22110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:10.533825 22110 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.190:35677
I20240501 14:00:10.533844 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.190:35677 every 8 connection(s)
I20240501 14:00:10.535281 22110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:00:10.539183 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.540546 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22110
I20240501 14:00:10.540777 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:00:10.552186 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad: Bootstrap starting.
I20240501 14:00:10.556059 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.557229 22176 log.cc:824] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:10.559952 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad: No bootstrap required, opened a new log
I20240501 14:00:10.566181 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95d660c7bd7435fa08604fea52664ad" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 35677 } }
I20240501 14:00:10.566581 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.566746 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95d660c7bd7435fa08604fea52664ad, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.567520 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95d660c7bd7435fa08604fea52664ad" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 35677 } }
I20240501 14:00:10.567759 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:00:10.567912 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:00:10.568078 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.570262 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95d660c7bd7435fa08604fea52664ad" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 35677 } }
I20240501 14:00:10.570894 22176 leader_election.cc:302] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e95d660c7bd7435fa08604fea52664ad; no voters:
I20240501 14:00:10.571553 22176 leader_election.cc:288] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:00:10.571728 22181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:10.572418 22181 raft_consensus.cc:686] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [term 1 LEADER]: Becoming Leader. State: Replica: e95d660c7bd7435fa08604fea52664ad, State: Running, Role: LEADER
I20240501 14:00:10.573088 22181 consensus_queue.cc:237] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95d660c7bd7435fa08604fea52664ad" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 35677 } }
I20240501 14:00:10.573899 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:00:10.576516 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e95d660c7bd7435fa08604fea52664ad. Latest consensus state: current_term: 1 leader_uuid: "e95d660c7bd7435fa08604fea52664ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95d660c7bd7435fa08604fea52664ad" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 35677 } } }
I20240501 14:00:10.576608 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e95d660c7bd7435fa08604fea52664ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95d660c7bd7435fa08604fea52664ad" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 35677 } } }
I20240501 14:00:10.576967 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:10.576974 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:10.578823 22187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:00:10.584847 22187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:00:10.594692 22187 catalog_manager.cc:1347] Generated new cluster ID: cd3946ada9374c5286a27e2c731c32b3
I20240501 14:00:10.594828 22187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:00:10.630045 22187 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:00:10.631326 22187 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:00:10.640535 22187 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e95d660c7bd7435fa08604fea52664ad: Generated new TSK 0
I20240501 14:00:10.641599 22187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:00:10.655341 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.129:0
--local_ip_for_outbound_sockets=127.16.36.129
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:00:10.995501 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:10.995842 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:10.996028 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:11.003780 22200 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:11.003911 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:11.004068 22200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.129
I20240501 14:00:11.013813 22200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.129
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:11.015293 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:11.017104 22200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:11.030776 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:11.031512 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:11.032289 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:11.033012 22200 server_base.cc:1034] running on GCE node
I20240501 14:00:11.033696 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:11.034796 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:11.036049 22200 hybrid_clock.cc:648] HybridClock initialized: now 1714572011035996 us; error 50 us; skew 500 ppm
I20240501 14:00:11.036387 22200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:11.038812 22200 webserver.cc:460] Webserver started at http://127.16.36.129:39791/ using document root <none> and password file <none>
I20240501 14:00:11.039584 22200 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:11.039710 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:11.040037 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:11.042863 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "474d063159a143fcba0416fa01328bc0"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:11.043696 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "474d063159a143fcba0416fa01328bc0"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:11.049146 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:00:11.053040 22214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:11.054664 22200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:11.054845 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "474d063159a143fcba0416fa01328bc0"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:11.055080 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:11.085462 22200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:11.086333 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:11.086583 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:11.087579 22200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:11.089083 22200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:11.089169 22200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:11.089255 22200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:11.089313 22200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:11.121877 22200 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.129:33963
I20240501 14:00:11.121941 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.129:33963 every 8 connection(s)
I20240501 14:00:11.123387 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:00:11.125699 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22200
I20240501 14:00:11.125962 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:00:11.129923 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.130:0
--local_ip_for_outbound_sockets=127.16.36.130
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:11.140658 22327 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:11.141069 22327 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:11.141934 22327 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:11.144330 22141 ts_manager.cc:194] Registered new tserver with Master: 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:11.145846 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.129:37417
W20240501 14:00:11.473915 22331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:11.474236 22331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:11.474403 22331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:11.482215 22331 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:11.482348 22331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:11.482475 22331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.130
I20240501 14:00:11.492205 22331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.130
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:11.493526 22331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:11.495395 22331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:11.508419 22336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:11.509307 22337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:11.510047 22339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:11.510743 22331 server_base.cc:1034] running on GCE node
I20240501 14:00:11.511392 22331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:11.512411 22331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:11.513643 22331 hybrid_clock.cc:648] HybridClock initialized: now 1714572011513589 us; error 45 us; skew 500 ppm
I20240501 14:00:11.513983 22331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:11.516391 22331 webserver.cc:460] Webserver started at http://127.16.36.130:34685/ using document root <none> and password file <none>
I20240501 14:00:11.517163 22331 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:11.517287 22331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:11.517606 22331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:11.520402 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:11.521253 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:11.526803 22331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 14:00:11.530527 22345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:11.532011 22331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:11.532191 22331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:11.532398 22331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:11.554796 22331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:11.555778 22331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:11.556041 22331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:11.557050 22331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:11.558681 22331 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:11.558773 22331 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:11.558864 22331 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:11.558948 22331 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:11.595595 22331 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.130:46541
I20240501 14:00:11.595647 22457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.130:46541 every 8 connection(s)
I20240501 14:00:11.597085 22331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:00:11.602538 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22331
I20240501 14:00:11.602741 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:00:11.606585 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.131:0
--local_ip_for_outbound_sockets=127.16.36.131
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:11.611169 22458 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:11.611624 22458 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:11.612452 22458 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:11.613946 22141 ts_manager.cc:194] Registered new tserver with Master: a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:11.615144 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.130:37699
W20240501 14:00:11.955996 22461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:11.956323 22461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:11.956459 22461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:11.964493 22461 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:11.964644 22461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:11.964834 22461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.131
I20240501 14:00:11.974449 22461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.131
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:11.975972 22461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:11.977855 22461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:11.992046 22466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:11.992473 22467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:11.993382 22469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:11.994287 22461 server_base.cc:1034] running on GCE node
I20240501 14:00:11.995083 22461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:11.996208 22461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:11.997442 22461 hybrid_clock.cc:648] HybridClock initialized: now 1714572011997389 us; error 39 us; skew 500 ppm
I20240501 14:00:11.997812 22461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:12.000448 22461 webserver.cc:460] Webserver started at http://127.16.36.131:34879/ using document root <none> and password file <none>
I20240501 14:00:12.001291 22461 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:12.001411 22461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:12.001763 22461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:12.004865 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:12.005795 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:12.011754 22461 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240501 14:00:12.015573 22475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.017210 22461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:12.017371 22461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
format_stamp: "Formatted at 2024-05-01 14:00:11 on dist-test-slave-9q9t"
I20240501 14:00:12.017587 22461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:12.047566 22461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:12.048524 22461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:12.048804 22461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:12.049818 22461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:12.051532 22461 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:12.051632 22461 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.051719 22461 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:12.051782 22461 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.085085 22461 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.131:38305
I20240501 14:00:12.085196 22587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.131:38305 every 8 connection(s)
I20240501 14:00:12.086534 22461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:00:12.087832 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22461
I20240501 14:00:12.088088 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:00:12.092214 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:0
--local_ip_for_outbound_sockets=127.16.36.132
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:12.102193 22588 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:12.102571 22588 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:12.103622 22588 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:12.105329 22141 ts_manager.cc:194] Registered new tserver with Master: 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:12.106206 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.131:45741
I20240501 14:00:12.148950 22327 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
W20240501 14:00:12.435688 22591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:12.436031 22591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:12.436206 22591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:12.444029 22591 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:12.444196 22591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:12.444363 22591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 14:00:12.453889 22591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:12.455333 22591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:12.457154 22591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:12.470508 22597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:12.471242 22596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:12.471746 22599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:12.473289 22591 server_base.cc:1034] running on GCE node
I20240501 14:00:12.473992 22591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:12.475117 22591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:12.476334 22591 hybrid_clock.cc:648] HybridClock initialized: now 1714572012476292 us; error 58 us; skew 500 ppm
I20240501 14:00:12.476665 22591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:12.479212 22591 webserver.cc:460] Webserver started at http://127.16.36.132:37233/ using document root <none> and password file <none>
I20240501 14:00:12.480072 22591 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:12.480185 22591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:12.480499 22591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:12.483330 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "90f980f10dfd441289a8ad090e23e200"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:12.484202 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "90f980f10dfd441289a8ad090e23e200"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:12.489925 22591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:00:12.493584 22605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.495276 22591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:12.495450 22591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "90f980f10dfd441289a8ad090e23e200"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:12.495666 22591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:12.522166 22591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:12.523221 22591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:12.523512 22591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:12.524525 22591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:12.526106 22591 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:12.526206 22591 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.526331 22591 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:12.526403 22591 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.560551 22591 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:45725
I20240501 14:00:12.560648 22717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:45725 every 8 connection(s)
I20240501 14:00:12.561933 22591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:00:12.563887 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22591
I20240501 14:00:12.564118 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:00:12.567440 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:0
--local_ip_for_outbound_sockets=127.16.36.133
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:12.577659 22718 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:12.577998 22718 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:12.578868 22718 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:12.580454 22141 ts_manager.cc:194] Registered new tserver with Master: 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725)
I20240501 14:00:12.581352 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:41641
I20240501 14:00:12.617856 22458 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
W20240501 14:00:12.913259 22721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:12.913602 22721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:12.913738 22721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:12.921375 22721 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:12.921533 22721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:12.921697 22721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 14:00:12.931039 22721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:12.932410 22721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:12.934177 22721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:12.947831 22729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:12.948557 22727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:12.949271 22726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:12.949913 22721 server_base.cc:1034] running on GCE node
I20240501 14:00:12.950546 22721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:12.951588 22721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:12.952833 22721 hybrid_clock.cc:648] HybridClock initialized: now 1714572012952761 us; error 60 us; skew 500 ppm
I20240501 14:00:12.953229 22721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:12.955708 22721 webserver.cc:460] Webserver started at http://127.16.36.133:33409/ using document root <none> and password file <none>
I20240501 14:00:12.956490 22721 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:12.956593 22721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:12.956873 22721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:12.959779 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5e1c3e3144384f90b47255623d63575b"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:12.960594 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5e1c3e3144384f90b47255623d63575b"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:12.966564 22721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20240501 14:00:12.970424 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.971979 22721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:12.972163 22721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5e1c3e3144384f90b47255623d63575b"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:12.972396 22721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:12.986958 22721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:12.987941 22721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:12.988236 22721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:12.989208 22721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:12.990819 22721 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:12.990923 22721 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:12.991047 22721 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:12.991122 22721 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:13.023862 22721 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:39481
I20240501 14:00:13.023921 22847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:39481 every 8 connection(s)
I20240501 14:00:13.025202 22721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:00:13.029007 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22721
I20240501 14:00:13.029268 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:00:13.032723 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.134:0
--local_ip_for_outbound_sockets=127.16.36.134
--webserver_interface=127.16.36.134
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:13.039692 22848 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:13.040055 22848 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:13.041014 22848 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:13.042564 22141 ts_manager.cc:194] Registered new tserver with Master: 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481)
I20240501 14:00:13.043442 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:54765
I20240501 14:00:13.109889 22588 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
W20240501 14:00:13.380705 22851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:13.381021 22851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:13.381134 22851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:13.388664 22851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:13.388794 22851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:13.388994 22851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.134
I20240501 14:00:13.398455 22851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.134
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.134
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:13.399801 22851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:13.401527 22851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:13.414533 22857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:13.415081 22859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:13.415618 22856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:13.416657 22851 server_base.cc:1034] running on GCE node
I20240501 14:00:13.417361 22851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:13.418407 22851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:13.419669 22851 hybrid_clock.cc:648] HybridClock initialized: now 1714572013419614 us; error 47 us; skew 500 ppm
I20240501 14:00:13.420053 22851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:13.422389 22851 webserver.cc:460] Webserver started at http://127.16.36.134:39539/ using document root <none> and password file <none>
I20240501 14:00:13.423229 22851 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:13.423352 22851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:13.423668 22851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:13.426326 22851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "dedba8a189394baca3188f2a5eddbcd4"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:13.427230 22851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "dedba8a189394baca3188f2a5eddbcd4"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:13.432947 22851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:00:13.436743 22865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:13.438310 22851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:13.438485 22851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "dedba8a189394baca3188f2a5eddbcd4"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:13.438736 22851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:13.460265 22851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:13.461169 22851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:13.461505 22851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:13.462494 22851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:13.464148 22851 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:13.464241 22851 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:13.464323 22851 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:13.464377 22851 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:13.497625 22851 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.134:35555
I20240501 14:00:13.497674 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.134:35555 every 8 connection(s)
I20240501 14:00:13.498991 22851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:00:13.504516 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22851
I20240501 14:00:13.504714 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:00:13.508170 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.135:0
--local_ip_for_outbound_sockets=127.16.36.135
--webserver_interface=127.16.36.135
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:13.513760 22978 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:13.514096 22978 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:13.514930 22978 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:13.516451 22141 ts_manager.cc:194] Registered new tserver with Master: dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555)
I20240501 14:00:13.517699 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.134:56845
I20240501 14:00:13.584044 22718 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
W20240501 14:00:13.852967 22981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:13.853304 22981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:13.853430 22981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:13.861313 22981 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:13.861442 22981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:13.861567 22981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.135
I20240501 14:00:13.870934 22981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.135
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.135
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:13.872262 22981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:13.874043 22981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:13.887184 22986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:13.888082 22987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:13.888734 22989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:13.889360 22981 server_base.cc:1034] running on GCE node
I20240501 14:00:13.890017 22981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:13.891099 22981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:13.892329 22981 hybrid_clock.cc:648] HybridClock initialized: now 1714572013892276 us; error 45 us; skew 500 ppm
I20240501 14:00:13.892691 22981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:13.895876 22981 webserver.cc:460] Webserver started at http://127.16.36.135:45885/ using document root <none> and password file <none>
I20240501 14:00:13.896996 22981 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:13.897166 22981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:13.897596 22981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:13.901417 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "bcf471e1f6eb42b8a9ae457e6239b89f"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:13.902359 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "bcf471e1f6eb42b8a9ae457e6239b89f"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:13.907897 22981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 14:00:13.911576 22995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:13.913024 22981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:13.913216 22981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "bcf471e1f6eb42b8a9ae457e6239b89f"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:13.913434 22981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:13.925781 22981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:13.926712 22981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:13.926997 22981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:13.928007 22981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:13.929509 22981 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:13.929611 22981 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:13.929729 22981 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:13.929802 22981 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:13.962353 22981 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.135:41675
I20240501 14:00:13.962411 23107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.135:41675 every 8 connection(s)
I20240501 14:00:13.963703 22981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:00:13.970758 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 22981
I20240501 14:00:13.971005 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:00:13.978487 23108 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:13.978852 23108 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:13.979691 23108 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:13.981225 22141 ts_manager.cc:194] Registered new tserver with Master: bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675)
I20240501 14:00:13.982045 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.135:52445
I20240501 14:00:13.988235 16530 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:00:14.004922 22141 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39352:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:00:14.046717 22848 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
I20240501 14:00:14.058049 22783 tablet_service.cc:1467] Processing CreateTablet for tablet ce58c7fa6c734d3fb453d132540f933c (DEFAULT_TABLE table=TestTable [id=11e8e30a8c6d4b28a2c2ac45c411c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.058933 23043 tablet_service.cc:1467] Processing CreateTablet for tablet ce58c7fa6c734d3fb453d132540f933c (DEFAULT_TABLE table=TestTable [id=11e8e30a8c6d4b28a2c2ac45c411c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.059873 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce58c7fa6c734d3fb453d132540f933c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.061003 23043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce58c7fa6c734d3fb453d132540f933c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.072784 23124 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Bootstrap starting.
I20240501 14:00:14.076848 23124 tablet_bootstrap.cc:654] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.078051 23125 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap starting.
I20240501 14:00:14.078375 23124 log.cc:824] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:14.083738 23125 tablet_bootstrap.cc:654] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.085279 23125 log.cc:824] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:14.089179 23124 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: No bootstrap required, opened a new log
I20240501 14:00:14.089323 23125 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: No bootstrap required, opened a new log
I20240501 14:00:14.089516 23124 ts_tablet_manager.cc:1367] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20240501 14:00:14.089627 23125 ts_tablet_manager.cc:1367] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 14:00:14.096808 23124 raft_consensus.cc:348] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:14.097347 23124 raft_consensus.cc:374] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.097471 23124 raft_consensus.cc:729] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.098281 23124 consensus_queue.cc:260] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:14.099963 23124 ts_tablet_manager.cc:1398] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20240501 14:00:14.099068 23125 raft_consensus.cc:348] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:14.100927 23125 raft_consensus.cc:374] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.101044 23125 raft_consensus.cc:729] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf471e1f6eb42b8a9ae457e6239b89f, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.101790 23125 consensus_queue.cc:260] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:14.103017 23108 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
I20240501 14:00:14.103660 23125 ts_tablet_manager.cc:1398] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20240501 14:00:14.215378 23109 tablet.cc:2369] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:00:14.277666 22849 tablet.cc:2369] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:14.360010 23128 raft_consensus.cc:482] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:14.360278 23128 raft_consensus.cc:504] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:14.361860 23128 leader_election.cc:288] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675)
I20240501 14:00:14.368355 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce58c7fa6c734d3fb453d132540f933c" candidate_uuid: "5e1c3e3144384f90b47255623d63575b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" is_pre_election: true
I20240501 14:00:14.368826 23063 raft_consensus.cc:2454] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c3e3144384f90b47255623d63575b in term 0.
I20240501 14:00:14.369412 22739 leader_election.cc:302] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5e1c3e3144384f90b47255623d63575b, bcf471e1f6eb42b8a9ae457e6239b89f; no voters:
I20240501 14:00:14.369783 23128 raft_consensus.cc:2781] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:14.369936 23128 raft_consensus.cc:482] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:14.370047 23128 raft_consensus.cc:3037] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.372128 23128 raft_consensus.cc:504] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:14.372618 23128 leader_election.cc:288] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 1 election: Requested vote from peers bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675)
I20240501 14:00:14.373176 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce58c7fa6c734d3fb453d132540f933c" candidate_uuid: "5e1c3e3144384f90b47255623d63575b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f"
I20240501 14:00:14.373348 23063 raft_consensus.cc:3037] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.375528 23063 raft_consensus.cc:2454] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c3e3144384f90b47255623d63575b in term 1.
I20240501 14:00:14.375927 22739 leader_election.cc:302] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5e1c3e3144384f90b47255623d63575b, bcf471e1f6eb42b8a9ae457e6239b89f; no voters:
I20240501 14:00:14.376225 23128 raft_consensus.cc:2781] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:14.376685 23128 raft_consensus.cc:686] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Running, Role: LEADER
I20240501 14:00:14.377192 23128 consensus_queue.cc:237] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:14.381866 22140 catalog_manager.cc:5484] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c3e3144384f90b47255623d63575b (127.16.36.133). New cstate: current_term: 1 leader_uuid: "5e1c3e3144384f90b47255623d63575b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:00:14.429505 16530 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:00:14.431883 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 474d063159a143fcba0416fa01328bc0 to finish bootstrapping
I20240501 14:00:14.441664 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a5574cf7d8b54da09d7f589c2c11e97c to finish bootstrapping
I20240501 14:00:14.451500 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3b1653dc15114e7c873f9cc1a9a3c0a5 to finish bootstrapping
I20240501 14:00:14.461014 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 90f980f10dfd441289a8ad090e23e200 to finish bootstrapping
I20240501 14:00:14.470345 16530 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5e1c3e3144384f90b47255623d63575b to finish bootstrapping
I20240501 14:00:14.478538 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dedba8a189394baca3188f2a5eddbcd4 to finish bootstrapping
I20240501 14:00:14.487749 16530 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bcf471e1f6eb42b8a9ae457e6239b89f to finish bootstrapping
I20240501 14:00:14.495406 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22591
I20240501 14:00:14.496515 22713 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:14.520516 22978 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
I20240501 14:00:14.580179 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22591
I20240501 14:00:14.598192 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22721
I20240501 14:00:14.599499 22843 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:14.691555 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22721
I20240501 14:00:14.708372 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22851
I20240501 14:00:14.709522 22973 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:14.791009 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22851
I20240501 14:00:14.807147 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22981
I20240501 14:00:14.808351 23103 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:14.894721 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22981
I20240501 14:00:18.664788 22140 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39352:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:18.689756 22393 tablet_service.cc:1467] Processing CreateTablet for tablet 161faaa672ce4a0e80495369f1060558 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.691421 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161faaa672ce4a0e80495369f1060558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.691035 22261 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebad65f05b645fdbd0844fee2b23dfe (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.692456 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebad65f05b645fdbd0844fee2b23dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.695003 22392 tablet_service.cc:1467] Processing CreateTablet for tablet 999759caca8344f2b2ae0cff5e204425 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.695580 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999759caca8344f2b2ae0cff5e204425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.696509 22262 tablet_service.cc:1467] Processing CreateTablet for tablet 161faaa672ce4a0e80495369f1060558 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.696955 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161faaa672ce4a0e80495369f1060558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.710366 23153 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:18.710105 22522 tablet_service.cc:1467] Processing CreateTablet for tablet 999759caca8344f2b2ae0cff5e204425 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.711982 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999759caca8344f2b2ae0cff5e204425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.713763 23153 tablet_bootstrap.cc:654] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.710143 22523 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebad65f05b645fdbd0844fee2b23dfe (DEFAULT_TABLE table=rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.714859 23153 log.cc:824] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:18.715370 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebad65f05b645fdbd0844fee2b23dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.720842 23154 tablet_bootstrap.cc:492] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:18.725741 23154 tablet_bootstrap.cc:654] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.727491 23153 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:18.727870 23153 ts_tablet_manager.cc:1367] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.011s
I20240501 14:00:18.728567 23154 log.cc:824] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:18.731699 23157 tablet_bootstrap.cc:492] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:18.734484 23157 tablet_bootstrap.cc:654] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.735075 23153 raft_consensus.cc:348] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:18.735455 23153 raft_consensus.cc:374] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.735571 23153 raft_consensus.cc:729] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.735875 23157 log.cc:824] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:18.736264 23153 consensus_queue.cc:260] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:18.736889 23154 tablet_bootstrap.cc:492] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:18.737275 23154 ts_tablet_manager.cc:1367] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20240501 14:00:18.737984 23153 ts_tablet_manager.cc:1398] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20240501 14:00:18.738373 23153 tablet_bootstrap.cc:492] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:18.740031 23153 tablet_bootstrap.cc:654] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.745265 23157 tablet_bootstrap.cc:492] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:18.745582 23157 ts_tablet_manager.cc:1367] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20240501 14:00:18.747824 23153 tablet_bootstrap.cc:492] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:18.747989 23153 ts_tablet_manager.cc:1367] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240501 14:00:18.748719 23153 raft_consensus.cc:348] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:18.748881 23153 raft_consensus.cc:374] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.748973 23153 raft_consensus.cc:729] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.748919 23154 raft_consensus.cc:348] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:18.749219 23153 consensus_queue.cc:260] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:18.749392 23154 raft_consensus.cc:374] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.749505 23154 raft_consensus.cc:729] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.749684 23153 ts_tablet_manager.cc:1398] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:18.750324 23154 consensus_queue.cc:260] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:18.752305 23154 ts_tablet_manager.cc:1398] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240501 14:00:18.752871 23154 tablet_bootstrap.cc:492] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:18.754709 23154 tablet_bootstrap.cc:654] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.754784 23157 raft_consensus.cc:348] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:18.755245 23157 raft_consensus.cc:374] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.755355 23157 raft_consensus.cc:729] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.756142 23157 consensus_queue.cc:260] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:18.756927 23154 tablet_bootstrap.cc:492] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:18.757112 23154 ts_tablet_manager.cc:1367] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:18.757820 23157 ts_tablet_manager.cc:1398] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20240501 14:00:18.758025 23154 raft_consensus.cc:348] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:18.758213 23154 raft_consensus.cc:374] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.758318 23157 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:18.758308 23154 raft_consensus.cc:729] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.758533 23154 consensus_queue.cc:260] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:18.759063 23154 ts_tablet_manager.cc:1398] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:18.760476 23157 tablet_bootstrap.cc:654] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.762356 23157 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:18.762492 23157 ts_tablet_manager.cc:1367] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:18.763446 23157 raft_consensus.cc:348] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:18.763598 23157 raft_consensus.cc:374] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.763697 23157 raft_consensus.cc:729] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.763900 23157 consensus_queue.cc:260] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:18.764307 23157 ts_tablet_manager.cc:1398] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:18.771935 23159 raft_consensus.cc:482] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.772188 23159 raft_consensus.cc:504] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:18.773736 23159 leader_election.cc:288] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:18.780293 22282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161faaa672ce4a0e80495369f1060558" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0" is_pre_election: true
I20240501 14:00:18.780797 22282 raft_consensus.cc:2454] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 0.
I20240501 14:00:18.781524 22346 leader_election.cc:302] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:18.781884 23159 raft_consensus.cc:2781] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.782042 23159 raft_consensus.cc:482] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.782146 23159 raft_consensus.cc:3037] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.784196 23159 raft_consensus.cc:504] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:18.784674 23159 leader_election.cc:288] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Requested vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:18.785216 22282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161faaa672ce4a0e80495369f1060558" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0"
I20240501 14:00:18.785377 22282 raft_consensus.cc:3037] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.787576 22282 raft_consensus.cc:2454] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 1.
I20240501 14:00:18.787982 22346 leader_election.cc:302] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:18.788280 23159 raft_consensus.cc:2781] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.788728 23159 raft_consensus.cc:686] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Leader. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:00:18.789229 23159 consensus_queue.cc:237] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:18.793285 22140 catalog_manager.cc:5484] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: term changed from 0 to 1, leader changed from <none> to a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130). New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } }
W20240501 14:00:18.844982 22589 tablet.cc:2369] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:00:18.854703 22459 tablet.cc:2369] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:00:18.881079 22328 tablet.cc:2369] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:18.921617 23163 raft_consensus.cc:482] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.921909 23163 raft_consensus.cc:504] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:18.923664 23163 leader_election.cc:288] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:18.930197 22282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebad65f05b645fdbd0844fee2b23dfe" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0" is_pre_election: true
I20240501 14:00:18.930691 22282 raft_consensus.cc:2454] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 0.
I20240501 14:00:18.931247 22476 leader_election.cc:302] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:18.931641 23163 raft_consensus.cc:2781] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.931782 23163 raft_consensus.cc:482] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.931876 23163 raft_consensus.cc:3037] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.933933 23163 raft_consensus.cc:504] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:18.934484 23163 leader_election.cc:288] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:18.935038 22282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebad65f05b645fdbd0844fee2b23dfe" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0"
I20240501 14:00:18.935256 22282 raft_consensus.cc:3037] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.937209 22282 raft_consensus.cc:2454] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 1.
I20240501 14:00:18.937613 22476 leader_election.cc:302] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:18.937906 23163 raft_consensus.cc:2781] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.938390 23163 raft_consensus.cc:686] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Running, Role: LEADER
I20240501 14:00:18.938867 23163 consensus_queue.cc:237] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:18.942932 22140 catalog_manager.cc:5484] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } }
I20240501 14:00:19.071432 23159 raft_consensus.cc:482] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.071697 23159 raft_consensus.cc:504] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:19.072351 23159 leader_election.cc:288] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:19.077944 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999759caca8344f2b2ae0cff5e204425" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" is_pre_election: true
I20240501 14:00:19.078347 22543 raft_consensus.cc:2454] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 0.
I20240501 14:00:19.078889 22346 leader_election.cc:302] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:19.079185 23159 raft_consensus.cc:2781] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.079281 23159 raft_consensus.cc:482] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.079394 23159 raft_consensus.cc:3037] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.081331 23159 raft_consensus.cc:504] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:19.081807 23159 leader_election.cc:288] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Requested vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:19.082306 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999759caca8344f2b2ae0cff5e204425" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
I20240501 14:00:19.082509 22543 raft_consensus.cc:3037] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.084656 22543 raft_consensus.cc:2454] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 1.
I20240501 14:00:19.085090 22346 leader_election.cc:302] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:19.085399 23159 raft_consensus.cc:2781] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:19.085594 23159 raft_consensus.cc:686] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Leader. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:00:19.085858 23159 consensus_queue.cc:237] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:19.088536 22140 catalog_manager.cc:5484] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: term changed from 0 to 1, leader changed from <none> to a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130). New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } }
I20240501 14:00:19.282001 23159 consensus_queue.cc:1035] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:19.490274 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:19.490684 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:19.498658 23172 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:19.503548 23180 consensus_queue.cc:1035] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 14:00:19.679646 22543 raft_consensus.cc:1261] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:19.680567 23159 consensus_queue.cc:1035] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 229.026 ms
time per row: 0.114513 ms
I20240501 14:00:19.878921 22136 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39352:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:19.888419 22392 tablet_service.cc:1467] Processing CreateTablet for tablet d4716a6313a541b981e5d27b01657956 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.888950 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4716a6313a541b981e5d27b01657956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.888856 22523 tablet_service.cc:1467] Processing CreateTablet for tablet dd5f6e034ac14cd2864d894fe5b00fd9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.889034 22262 tablet_service.cc:1467] Processing CreateTablet for tablet d4716a6313a541b981e5d27b01657956 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.889288 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5f6e034ac14cd2864d894fe5b00fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.889365 22522 tablet_service.cc:1467] Processing CreateTablet for tablet 83ae513a62f345d1a6eac152a56936d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.889493 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4716a6313a541b981e5d27b01657956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.889719 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ae513a62f345d1a6eac152a56936d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.892711 22393 tablet_service.cc:1467] Processing CreateTablet for tablet 83ae513a62f345d1a6eac152a56936d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.894079 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ae513a62f345d1a6eac152a56936d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.898021 22261 tablet_service.cc:1467] Processing CreateTablet for tablet dd5f6e034ac14cd2864d894fe5b00fd9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.898584 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5f6e034ac14cd2864d894fe5b00fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.915261 23234 tablet_bootstrap.cc:492] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:19.917929 23232 tablet_bootstrap.cc:492] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:19.918131 23234 tablet_bootstrap.cc:654] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.919596 23233 tablet_bootstrap.cc:492] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:19.920141 23232 tablet_bootstrap.cc:654] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.921033 23233 tablet_bootstrap.cc:654] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.922251 23234 tablet_bootstrap.cc:492] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:19.922508 23234 ts_tablet_manager.cc:1367] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 14:00:19.923086 23232 tablet_bootstrap.cc:492] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:19.923255 23232 ts_tablet_manager.cc:1367] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:19.923594 23234 raft_consensus.cc:348] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:19.923812 23234 raft_consensus.cc:374] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.923918 23234 raft_consensus.cc:729] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.924144 23232 raft_consensus.cc:348] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.924176 23234 consensus_queue.cc:260] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:19.924362 23232 raft_consensus.cc:374] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.924465 23232 raft_consensus.cc:729] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.924795 23232 consensus_queue.cc:260] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.924932 23234 ts_tablet_manager.cc:1398] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:19.925225 23234 tablet_bootstrap.cc:492] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:19.926932 23234 tablet_bootstrap.cc:654] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.925791 23232 ts_tablet_manager.cc:1398] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:19.927242 23232 tablet_bootstrap.cc:492] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:19.928928 23232 tablet_bootstrap.cc:654] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.930361 23233 tablet_bootstrap.cc:492] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:19.930642 23233 ts_tablet_manager.cc:1367] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 14:00:19.931552 23233 raft_consensus.cc:348] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.931842 23233 raft_consensus.cc:374] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.932008 23233 raft_consensus.cc:729] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.932341 23233 consensus_queue.cc:260] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.933034 23233 ts_tablet_manager.cc:1398] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:19.933197 23234 tablet_bootstrap.cc:492] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:19.933394 23233 tablet_bootstrap.cc:492] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:19.933465 23234 ts_tablet_manager.cc:1367] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 14:00:19.934526 23234 raft_consensus.cc:348] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.934900 23234 raft_consensus.cc:374] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.935194 23233 tablet_bootstrap.cc:654] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.935042 23234 raft_consensus.cc:729] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.935968 23234 consensus_queue.cc:260] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.937177 23232 tablet_bootstrap.cc:492] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:19.937390 23232 ts_tablet_manager.cc:1367] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.009s
I20240501 14:00:19.937438 23234 ts_tablet_manager.cc:1398] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:19.938221 23232 raft_consensus.cc:348] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:19.938478 23232 raft_consensus.cc:374] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.938566 23232 raft_consensus.cc:729] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.938856 23232 consensus_queue.cc:260] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:19.939461 23232 ts_tablet_manager.cc:1398] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:19.940843 23233 tablet_bootstrap.cc:492] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:19.941012 23233 ts_tablet_manager.cc:1367] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 14:00:19.941702 23233 raft_consensus.cc:348] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.941884 23233 raft_consensus.cc:374] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.941958 23233 raft_consensus.cc:729] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.942198 23233 consensus_queue.cc:260] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.943100 23233 ts_tablet_manager.cc:1398] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:19.955230 23180 raft_consensus.cc:482] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.955463 23180 raft_consensus.cc:504] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.956113 23180 leader_election.cc:288] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:19.956745 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5f6e034ac14cd2864d894fe5b00fd9" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0" is_pre_election: true
I20240501 14:00:19.956951 22281 raft_consensus.cc:2454] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 0.
I20240501 14:00:19.957391 22476 leader_election.cc:302] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:19.957684 23180 raft_consensus.cc:2781] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.957777 23180 raft_consensus.cc:482] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.957855 23180 raft_consensus.cc:3037] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.959895 23180 raft_consensus.cc:504] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.960335 23180 leader_election.cc:288] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:19.960847 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5f6e034ac14cd2864d894fe5b00fd9" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0"
I20240501 14:00:19.961030 22281 raft_consensus.cc:3037] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.963075 22281 raft_consensus.cc:2454] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 1.
I20240501 14:00:19.963460 22476 leader_election.cc:302] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:19.963763 23180 raft_consensus.cc:2781] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:19.963910 23180 raft_consensus.cc:686] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Running, Role: LEADER
I20240501 14:00:19.964164 23180 consensus_queue.cc:237] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.966881 22134 catalog_manager.cc:5484] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } }
I20240501 14:00:19.985870 23173 raft_consensus.cc:482] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.986106 23173 raft_consensus.cc:504] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.987607 23173 leader_election.cc:288] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:19.994016 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4716a6313a541b981e5d27b01657956" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" is_pre_election: true
I20240501 14:00:19.994462 22413 raft_consensus.cc:2454] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 0.
I20240501 14:00:19.995060 22218 leader_election.cc:302] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:19.995494 23173 raft_consensus.cc:2781] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.995641 23173 raft_consensus.cc:482] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.995741 23173 raft_consensus.cc:3037] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.997817 23173 raft_consensus.cc:504] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:19.998364 23173 leader_election.cc:288] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Requested vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:19.998981 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4716a6313a541b981e5d27b01657956" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
I20240501 14:00:19.999162 22413 raft_consensus.cc:3037] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.001303 22413 raft_consensus.cc:2454] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 1.
I20240501 14:00:20.001777 22218 leader_election.cc:302] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:20.002100 23173 raft_consensus.cc:2781] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:20.002553 23173 raft_consensus.cc:686] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Running, Role: LEADER
I20240501 14:00:20.003113 23173 consensus_queue.cc:237] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:20.006294 22134 catalog_manager.cc:5484] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 474d063159a143fcba0416fa01328bc0 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } }
I20240501 14:00:20.207527 23166 raft_consensus.cc:482] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:20.207732 23166 raft_consensus.cc:504] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:20.208348 23166 leader_election.cc:288] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:20.209059 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ae513a62f345d1a6eac152a56936d0" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" is_pre_election: true
I20240501 14:00:20.209306 22543 raft_consensus.cc:2454] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 0.
I20240501 14:00:20.209916 22346 leader_election.cc:302] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:20.210280 23166 raft_consensus.cc:2781] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:20.210420 23166 raft_consensus.cc:482] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:20.210532 23166 raft_consensus.cc:3037] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.212554 23166 raft_consensus.cc:504] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:20.213069 23166 leader_election.cc:288] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Requested vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:20.213786 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ae513a62f345d1a6eac152a56936d0" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
I20240501 14:00:20.214051 22543 raft_consensus.cc:3037] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.216068 22543 raft_consensus.cc:2454] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 1.
I20240501 14:00:20.216526 22346 leader_election.cc:302] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:20.216861 23166 raft_consensus.cc:2781] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:20.217044 23166 raft_consensus.cc:686] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Leader. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:00:20.217340 23166 consensus_queue.cc:237] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:20.220005 22134 catalog_manager.cc:5484] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: term changed from 0 to 1, leader changed from <none> to a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130). New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } }
I20240501 14:00:20.439728 23173 consensus_queue.cc:1035] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.518139 23180 consensus_queue.cc:1035] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:20.622462 23246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:20.622866 23246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:20.630784 23246 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:00:20.776916 22543 raft_consensus.cc:1261] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:20.777736 23166 consensus_queue.cc:1035] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.812085 23286 mvcc.cc:205] Tried to move back new op lower bound from 7022886997073952768 to 7022886994811879424. Current Snapshot: MvccSnapshot[applied={T|T < 7022886997073952768}]
INSERT report
rows total: 2000
time total: 176.185 ms
time per row: 0.0880926 ms
I20240501 14:00:20.954598 22133 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39352:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:20.964037 22262 tablet_service.cc:1467] Processing CreateTablet for tablet dc19025daec44883afec64b3d12c7c4e (DEFAULT_TABLE table=rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.964183 22392 tablet_service.cc:1467] Processing CreateTablet for tablet dc19025daec44883afec64b3d12c7c4e (DEFAULT_TABLE table=rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.964571 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19025daec44883afec64b3d12c7c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.964644 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19025daec44883afec64b3d12c7c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.969287 23300 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:20.969321 23301 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:20.969820 22522 tablet_service.cc:1467] Processing CreateTablet for tablet a40334656b4b4333a13956fec5a1053a (DEFAULT_TABLE table=rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.970340 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40334656b4b4333a13956fec5a1053a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.970718 22393 tablet_service.cc:1467] Processing CreateTablet for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf (DEFAULT_TABLE table=rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.971119 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.971227 23301 tablet_bootstrap.cc:654] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.971305 23300 tablet_bootstrap.cc:654] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.971731 22523 tablet_service.cc:1467] Processing CreateTablet for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf (DEFAULT_TABLE table=rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.972175 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.975628 22261 tablet_service.cc:1467] Processing CreateTablet for tablet a40334656b4b4333a13956fec5a1053a (DEFAULT_TABLE table=rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:20.976183 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40334656b4b4333a13956fec5a1053a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.986222 23301 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:20.986498 23301 ts_tablet_manager.cc:1367] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20240501 14:00:20.986721 23300 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:20.986961 23300 ts_tablet_manager.cc:1367] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20240501 14:00:20.987530 23301 raft_consensus.cc:348] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:20.987861 23301 raft_consensus.cc:374] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.987862 23300 raft_consensus.cc:348] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:20.988055 23301 raft_consensus.cc:729] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.988181 23300 raft_consensus.cc:374] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.988293 23300 raft_consensus.cc:729] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.988478 23301 consensus_queue.cc:260] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:20.988535 23300 consensus_queue.cc:260] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:20.989212 23301 ts_tablet_manager.cc:1398] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:20.990417 23300 ts_tablet_manager.cc:1398] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:20.995149 23300 tablet_bootstrap.cc:492] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:20.995750 23301 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:20.996089 23304 tablet_bootstrap.cc:492] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:20.997265 23300 tablet_bootstrap.cc:654] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.997673 23304 tablet_bootstrap.cc:654] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.997699 23301 tablet_bootstrap.cc:654] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.999539 23300 tablet_bootstrap.cc:492] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:20.999688 23300 ts_tablet_manager.cc:1367] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:20.999926 23304 tablet_bootstrap.cc:492] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:21.000077 23304 ts_tablet_manager.cc:1367] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:00:21.000475 23301 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:21.000504 23300 raft_consensus.cc:348] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:21.000648 23301 ts_tablet_manager.cc:1367] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:21.000730 23300 raft_consensus.cc:374] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.000819 23300 raft_consensus.cc:729] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.000748 23304 raft_consensus.cc:348] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:21.000991 23304 raft_consensus.cc:374] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.001101 23304 raft_consensus.cc:729] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.001062 23300 consensus_queue.cc:260] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:21.001399 23304 consensus_queue.cc:260] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:21.001430 23301 raft_consensus.cc:348] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.001627 23301 raft_consensus.cc:374] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.001637 23300 ts_tablet_manager.cc:1398] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:21.001701 23301 raft_consensus.cc:729] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.001894 23304 ts_tablet_manager.cc:1398] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:21.001927 23301 consensus_queue.cc:260] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.002192 23304 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:21.002403 23301 ts_tablet_manager.cc:1398] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:21.003821 23304 tablet_bootstrap.cc:654] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.006225 23304 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:21.006403 23304 ts_tablet_manager.cc:1367] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:21.007333 23304 raft_consensus.cc:348] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.007550 23304 raft_consensus.cc:374] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.007634 23304 raft_consensus.cc:729] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.007894 23304 consensus_queue.cc:260] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.008477 23304 ts_tablet_manager.cc:1398] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:21.088002 23289 raft_consensus.cc:482] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:21.088234 23289 raft_consensus.cc:504] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.088927 23289 leader_election.cc:288] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:21.089545 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0" is_pre_election: true
I20240501 14:00:21.089803 22281 raft_consensus.cc:2454] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 0.
I20240501 14:00:21.090284 22346 leader_election.cc:302] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:21.090571 23289 raft_consensus.cc:2781] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:21.090727 23289 raft_consensus.cc:482] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:21.090822 23289 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.092984 23289 raft_consensus.cc:504] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.093468 23289 leader_election.cc:288] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Requested vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:21.094065 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0"
I20240501 14:00:21.094302 22281 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.097160 22281 raft_consensus.cc:2454] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 1.
I20240501 14:00:21.097626 22346 leader_election.cc:302] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:21.097991 23289 raft_consensus.cc:2781] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:21.098165 23289 raft_consensus.cc:686] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Leader. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:00:21.098511 23289 consensus_queue.cc:237] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.101579 22133 catalog_manager.cc:5484] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: term changed from 0 to 1, leader changed from <none> to a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130). New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } }
I20240501 14:00:21.196537 23243 raft_consensus.cc:482] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:21.196743 23243 raft_consensus.cc:504] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:21.197428 23243 leader_election.cc:288] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:21.203490 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40334656b4b4333a13956fec5a1053a" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" is_pre_election: true
I20240501 14:00:21.203752 22543 raft_consensus.cc:2454] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 0.
I20240501 14:00:21.204221 22215 leader_election.cc:302] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:21.204555 23243 raft_consensus.cc:2781] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:21.204671 23243 raft_consensus.cc:482] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:21.204766 23243 raft_consensus.cc:3037] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.206888 23243 raft_consensus.cc:504] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:21.207399 23243 leader_election.cc:288] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Requested vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:21.208099 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40334656b4b4333a13956fec5a1053a" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
I20240501 14:00:21.208331 22543 raft_consensus.cc:3037] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.210299 22543 raft_consensus.cc:2454] T a40334656b4b4333a13956fec5a1053a P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 1.
I20240501 14:00:21.210932 22215 leader_election.cc:302] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:21.211359 23243 raft_consensus.cc:2781] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:21.211512 23243 raft_consensus.cc:686] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Running, Role: LEADER
I20240501 14:00:21.211822 23243 consensus_queue.cc:237] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:21.215534 22133 catalog_manager.cc:5484] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 474d063159a143fcba0416fa01328bc0 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } }
I20240501 14:00:21.333719 23289 raft_consensus.cc:482] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:21.333973 23289 raft_consensus.cc:504] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.334542 23289 leader_election.cc:288] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:21.335186 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09f45be8c064ecdbdc78d5ce8dbe8bf" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" is_pre_election: true
I20240501 14:00:21.335448 22543 raft_consensus.cc:2454] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 0.
I20240501 14:00:21.335887 22346 leader_election.cc:302] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:21.336208 23289 raft_consensus.cc:2781] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:21.336305 23289 raft_consensus.cc:482] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:21.336417 23289 raft_consensus.cc:3037] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.338384 23289 raft_consensus.cc:504] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.338912 23289 leader_election.cc:288] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Requested vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:21.339458 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09f45be8c064ecdbdc78d5ce8dbe8bf" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
I20240501 14:00:21.339635 22543 raft_consensus.cc:3037] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:21.341485 22543 raft_consensus.cc:2454] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 1.
I20240501 14:00:21.341879 22346 leader_election.cc:302] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:21.342140 23289 raft_consensus.cc:2781] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:21.342298 23289 raft_consensus.cc:686] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Leader. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:00:21.342572 23289 consensus_queue.cc:237] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:21.345160 22133 catalog_manager.cc:5484] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: term changed from 0 to 1, leader changed from <none> to a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130). New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } }
I20240501 14:00:21.539970 23289 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:21.614861 23243 consensus_queue.cc:1035] T a40334656b4b4333a13956fec5a1053a P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:21.713387 23313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:21.713789 23313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:21.721608 23313 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:00:21.868839 22543 raft_consensus.cc:1261] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:21.869524 23293 consensus_queue.cc:1035] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:21.898979 23315 mvcc.cc:205] Tried to move back new op lower bound from 7022887001566920704 to 7022886999421054976. Current Snapshot: MvccSnapshot[applied={T|T < 7022887001566920704}]
INSERT report
rows total: 2000
time total: 184.805 ms
time per row: 0.0924025 ms
I20240501 14:00:22.055318 22140 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39352:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:22.064921 22261 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcbcadfd9a34282950350f04b7fcce5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.065016 22262 tablet_service.cc:1467] Processing CreateTablet for tablet c40fd7b8bfd04d8ca67a9acf40f74445 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.065099 22522 tablet_service.cc:1467] Processing CreateTablet for tablet c40fd7b8bfd04d8ca67a9acf40f74445 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.065140 22393 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcbcadfd9a34282950350f04b7fcce5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.065553 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcbcadfd9a34282950350f04b7fcce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.065577 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40fd7b8bfd04d8ca67a9acf40f74445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.065577 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcbcadfd9a34282950350f04b7fcce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.070514 23371 tablet_bootstrap.cc:492] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:22.070592 22523 tablet_service.cc:1467] Processing CreateTablet for tablet 74e03b7d269c40ff9fbf6457611d849b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.071080 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e03b7d269c40ff9fbf6457611d849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.072566 23371 tablet_bootstrap.cc:654] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.072839 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40fd7b8bfd04d8ca67a9acf40f74445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.076095 22392 tablet_service.cc:1467] Processing CreateTablet for tablet 74e03b7d269c40ff9fbf6457611d849b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:22.076705 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e03b7d269c40ff9fbf6457611d849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:22.080767 23373 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:22.083456 23373 tablet_bootstrap.cc:654] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.089049 23372 tablet_bootstrap.cc:492] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:22.096504 23372 tablet_bootstrap.cc:654] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.099386 23373 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:22.101743 23373 ts_tablet_manager.cc:1367] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20240501 14:00:22.103039 23373 raft_consensus.cc:348] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:22.103521 23373 raft_consensus.cc:374] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.103729 23373 raft_consensus.cc:729] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.104175 23373 consensus_queue.cc:260] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:22.105180 23373 ts_tablet_manager.cc:1398] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:22.105636 23373 tablet_bootstrap.cc:492] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:22.107709 23372 tablet_bootstrap.cc:492] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:22.107923 23372 ts_tablet_manager.cc:1367] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20240501 14:00:22.110682 23372 raft_consensus.cc:348] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:22.111152 23372 raft_consensus.cc:374] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.111289 23372 raft_consensus.cc:729] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.111536 23372 consensus_queue.cc:260] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:22.112857 23371 tablet_bootstrap.cc:492] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:22.112987 23372 ts_tablet_manager.cc:1398] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:22.113029 23371 ts_tablet_manager.cc:1367] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.020s
I20240501 14:00:22.113277 23372 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:22.113921 23371 raft_consensus.cc:348] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:22.114145 23371 raft_consensus.cc:374] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.114254 23371 raft_consensus.cc:729] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.114500 23371 consensus_queue.cc:260] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:22.115149 23371 ts_tablet_manager.cc:1398] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:22.115532 23371 tablet_bootstrap.cc:492] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:22.115900 23372 tablet_bootstrap.cc:654] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.117187 23371 tablet_bootstrap.cc:654] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.114706 23373 tablet_bootstrap.cc:654] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:22.124840 23372 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:22.125042 23372 ts_tablet_manager.cc:1367] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240501 14:00:22.125850 23372 raft_consensus.cc:348] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:22.126083 23372 raft_consensus.cc:374] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.126159 23372 raft_consensus.cc:729] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.126430 23372 consensus_queue.cc:260] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:22.128263 23372 ts_tablet_manager.cc:1398] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:22.130975 23373 tablet_bootstrap.cc:492] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:22.131165 23373 ts_tablet_manager.cc:1367] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20240501 14:00:22.131421 23371 tablet_bootstrap.cc:492] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:22.131587 23371 ts_tablet_manager.cc:1367] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240501 14:00:22.132068 23373 raft_consensus.cc:348] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:22.132321 23373 raft_consensus.cc:374] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.132403 23373 raft_consensus.cc:729] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.132406 23371 raft_consensus.cc:348] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:22.132630 23371 raft_consensus.cc:374] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:22.132712 23371 raft_consensus.cc:729] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:22.132719 23373 consensus_queue.cc:260] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:22.132954 23371 consensus_queue.cc:260] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:22.133852 23371 ts_tablet_manager.cc:1398] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:22.134426 23373 ts_tablet_manager.cc:1398] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:00:22.173105 23251 raft_consensus.cc:482] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.173346 23251 raft_consensus.cc:504] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:22.174193 23251 leader_election.cc:288] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:22.181303 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e03b7d269c40ff9fbf6457611d849b" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" is_pre_election: true
I20240501 14:00:22.181586 22413 raft_consensus.cc:2454] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 0.
I20240501 14:00:22.182054 22479 leader_election.cc:302] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:22.182381 23251 raft_consensus.cc:2781] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.182495 23251 raft_consensus.cc:482] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.182562 23251 raft_consensus.cc:3037] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.184604 23251 raft_consensus.cc:504] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:22.185084 23251 leader_election.cc:288] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Requested vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:22.185654 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e03b7d269c40ff9fbf6457611d849b" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
I20240501 14:00:22.185873 22413 raft_consensus.cc:3037] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.187850 22413 raft_consensus.cc:2454] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 1.
I20240501 14:00:22.188335 22479 leader_election.cc:302] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:22.188671 23251 raft_consensus.cc:2781] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.189076 23251 raft_consensus.cc:686] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Running, Role: LEADER
I20240501 14:00:22.189394 23251 consensus_queue.cc:237] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:22.192034 22137 catalog_manager.cc:5484] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } }
I20240501 14:00:22.198899 23173 raft_consensus.cc:482] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.199122 23173 raft_consensus.cc:504] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:22.199685 23173 leader_election.cc:288] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:22.200310 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcbcadfd9a34282950350f04b7fcce5" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" is_pre_election: true
I20240501 14:00:22.200539 22413 raft_consensus.cc:2454] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 0.
I20240501 14:00:22.200954 22218 leader_election.cc:302] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:22.201251 23173 raft_consensus.cc:2781] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.201362 23173 raft_consensus.cc:482] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.201450 23173 raft_consensus.cc:3037] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.203492 23173 raft_consensus.cc:504] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:22.203994 23173 leader_election.cc:288] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Requested vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:22.204650 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcbcadfd9a34282950350f04b7fcce5" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
I20240501 14:00:22.204859 22413 raft_consensus.cc:3037] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.206868 22413 raft_consensus.cc:2454] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 1.
I20240501 14:00:22.207337 22218 leader_election.cc:302] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:22.207650 23173 raft_consensus.cc:2781] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.207798 23173 raft_consensus.cc:686] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Running, Role: LEADER
I20240501 14:00:22.208076 23173 consensus_queue.cc:237] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:22.211202 22137 catalog_manager.cc:5484] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 474d063159a143fcba0416fa01328bc0 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } }
I20240501 14:00:22.358124 23251 raft_consensus.cc:482] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.358359 23251 raft_consensus.cc:504] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:22.359012 23251 leader_election.cc:288] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:22.359666 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40fd7b8bfd04d8ca67a9acf40f74445" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0" is_pre_election: true
I20240501 14:00:22.359870 22281 raft_consensus.cc:2454] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 0.
I20240501 14:00:22.360309 22476 leader_election.cc:302] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:22.360632 23251 raft_consensus.cc:2781] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.360733 23251 raft_consensus.cc:482] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.360813 23251 raft_consensus.cc:3037] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.362768 23251 raft_consensus.cc:504] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:22.363291 23251 leader_election.cc:288] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:22.364225 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40fd7b8bfd04d8ca67a9acf40f74445" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0"
I20240501 14:00:22.364450 22281 raft_consensus.cc:3037] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.366540 22281 raft_consensus.cc:2454] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 1.
I20240501 14:00:22.367007 22476 leader_election.cc:302] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:22.367264 23251 raft_consensus.cc:2781] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.367429 23251 raft_consensus.cc:686] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Running, Role: LEADER
I20240501 14:00:22.367714 23251 consensus_queue.cc:237] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:22.370266 22137 catalog_manager.cc:5484] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } }
I20240501 14:00:22.672874 23251 consensus_queue.cc:1035] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:22.725363 23385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:22.725759 23385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:22.733465 23385 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:22.763520 23173 consensus_queue.cc:1035] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 14:00:22.890880 22281 raft_consensus.cc:1261] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:22.891718 23251 consensus_queue.cc:1035] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.926801 23425 mvcc.cc:205] Tried to move back new op lower bound from 7022887005752426496 to 7022887003619917824. Current Snapshot: MvccSnapshot[applied={T|T < 7022887005752426496}]
W20240501 14:00:22.975349 22218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:46541, user_credentials={real_user=slave}} blocked reactor thread for 53447us
INSERT report
rows total: 2000
time total: 211.983 ms
time per row: 0.105991 ms
I20240501 14:00:23.100304 22139 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39352:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:23.110340 22261 tablet_service.cc:1467] Processing CreateTablet for tablet fe968af0ff994a9093beca7a614e77b7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.110504 22392 tablet_service.cc:1467] Processing CreateTablet for tablet fe968af0ff994a9093beca7a614e77b7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.110719 22523 tablet_service.cc:1467] Processing CreateTablet for tablet e6ac874a7f084076bf03ed4167f967e0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.110986 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe968af0ff994a9093beca7a614e77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.111029 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe968af0ff994a9093beca7a614e77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.111153 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac874a7f084076bf03ed4167f967e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.115280 22393 tablet_service.cc:1467] Processing CreateTablet for tablet b28e7ed3a23243ee98327f60b2ebdbd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.115778 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28e7ed3a23243ee98327f60b2ebdbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.124341 22262 tablet_service.cc:1467] Processing CreateTablet for tablet e6ac874a7f084076bf03ed4167f967e0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.124893 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac874a7f084076bf03ed4167f967e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.126500 22522 tablet_service.cc:1467] Processing CreateTablet for tablet b28e7ed3a23243ee98327f60b2ebdbd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.127128 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28e7ed3a23243ee98327f60b2ebdbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.128338 23445 tablet_bootstrap.cc:492] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:23.131402 23444 tablet_bootstrap.cc:492] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:23.132814 23445 tablet_bootstrap.cc:654] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.133642 23444 tablet_bootstrap.cc:654] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.140992 23443 tablet_bootstrap.cc:492] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:23.142832 23443 tablet_bootstrap.cc:654] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.142899 23444 tablet_bootstrap.cc:492] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:23.143191 23444 ts_tablet_manager.cc:1367] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240501 14:00:23.144344 23444 raft_consensus.cc:348] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:23.144662 23444 raft_consensus.cc:374] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.144804 23444 raft_consensus.cc:729] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.145190 23444 consensus_queue.cc:260] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:23.145854 23444 ts_tablet_manager.cc:1398] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:23.146248 23444 tablet_bootstrap.cc:492] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0: Bootstrap starting.
I20240501 14:00:23.146262 23445 tablet_bootstrap.cc:492] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:23.146714 23445 ts_tablet_manager.cc:1367] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20240501 14:00:23.147703 23445 raft_consensus.cc:348] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:23.148149 23444 tablet_bootstrap.cc:654] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.148290 23445 raft_consensus.cc:374] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.148408 23445 raft_consensus.cc:729] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.148687 23445 consensus_queue.cc:260] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:23.149737 23445 ts_tablet_manager.cc:1398] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:23.150337 23445 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Bootstrap starting.
I20240501 14:00:23.153156 23445 tablet_bootstrap.cc:654] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.155452 23443 tablet_bootstrap.cc:492] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:23.155697 23443 ts_tablet_manager.cc:1367] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240501 14:00:23.155895 23445 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: No bootstrap required, opened a new log
I20240501 14:00:23.156201 23445 ts_tablet_manager.cc:1367] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 14:00:23.156517 23443 raft_consensus.cc:348] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:23.156772 23443 raft_consensus.cc:374] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.156898 23443 raft_consensus.cc:729] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.157160 23443 consensus_queue.cc:260] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:23.157456 23445 raft_consensus.cc:348] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:23.157759 23443 ts_tablet_manager.cc:1398] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:23.157812 23445 raft_consensus.cc:374] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.158008 23445 raft_consensus.cc:729] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.158087 23443 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: Bootstrap starting.
I20240501 14:00:23.160552 23443 tablet_bootstrap.cc:654] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.160893 23445 consensus_queue.cc:260] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:23.162328 23445 ts_tablet_manager.cc:1398] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:00:23.172006 23443 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: No bootstrap required, opened a new log
I20240501 14:00:23.172037 23444 tablet_bootstrap.cc:492] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0: No bootstrap required, opened a new log
I20240501 14:00:23.172215 23444 ts_tablet_manager.cc:1367] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.010s
I20240501 14:00:23.172210 23443 ts_tablet_manager.cc:1367] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20240501 14:00:23.172933 23443 raft_consensus.cc:348] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:23.173137 23443 raft_consensus.cc:374] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.173034 23444 raft_consensus.cc:348] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:23.173246 23444 raft_consensus.cc:374] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.173256 23443 raft_consensus.cc:729] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.173372 23444 raft_consensus.cc:729] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.173532 23443 consensus_queue.cc:260] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:23.173660 23444 consensus_queue.cc:260] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:23.174173 23443 ts_tablet_manager.cc:1398] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:23.174265 23444 ts_tablet_manager.cc:1398] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:00:23.301388 23357 raft_consensus.cc:482] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.301636 23357 raft_consensus.cc:504] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:23.302299 23357 leader_election.cc:288] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:23.302994 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe968af0ff994a9093beca7a614e77b7" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0" is_pre_election: true
I20240501 14:00:23.303218 22281 raft_consensus.cc:2454] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 0.
I20240501 14:00:23.303685 22346 leader_election.cc:302] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:23.303975 23357 raft_consensus.cc:2781] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.304096 23357 raft_consensus.cc:482] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.304175 23357 raft_consensus.cc:3037] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.306102 23357 raft_consensus.cc:504] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:23.306613 23357 leader_election.cc:288] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Requested vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:23.307299 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe968af0ff994a9093beca7a614e77b7" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474d063159a143fcba0416fa01328bc0"
I20240501 14:00:23.307485 22281 raft_consensus.cc:3037] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.309410 22281 raft_consensus.cc:2454] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 1.
I20240501 14:00:23.309862 22346 leader_election.cc:302] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:23.310184 23357 raft_consensus.cc:2781] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.310582 23357 raft_consensus.cc:686] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Leader. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:00:23.310904 23357 consensus_queue.cc:237] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } }
I20240501 14:00:23.313725 22140 catalog_manager.cc:5484] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: term changed from 0 to 1, leader changed from <none> to a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130). New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } }
I20240501 14:00:23.354269 23399 raft_consensus.cc:482] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.354496 23399 raft_consensus.cc:504] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:23.355127 23399 leader_election.cc:288] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:23.355808 22542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac874a7f084076bf03ed4167f967e0" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" is_pre_election: true
I20240501 14:00:23.356043 22542 raft_consensus.cc:2454] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 0.
I20240501 14:00:23.356572 22215 leader_election.cc:302] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:23.356861 23399 raft_consensus.cc:2781] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.356987 23399 raft_consensus.cc:482] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.357084 23399 raft_consensus.cc:3037] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.359136 23399 raft_consensus.cc:504] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:23.359625 23399 leader_election.cc:288] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Requested vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:23.360220 22542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac874a7f084076bf03ed4167f967e0" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
I20240501 14:00:23.360419 22542 raft_consensus.cc:3037] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.362411 22542 raft_consensus.cc:2454] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 1.
I20240501 14:00:23.362877 22215 leader_election.cc:302] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, 474d063159a143fcba0416fa01328bc0; no voters:
I20240501 14:00:23.363217 23399 raft_consensus.cc:2781] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.363363 23399 raft_consensus.cc:686] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Running, Role: LEADER
I20240501 14:00:23.363654 23399 consensus_queue.cc:237] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } }
I20240501 14:00:23.366400 22140 catalog_manager.cc:5484] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 474d063159a143fcba0416fa01328bc0 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } }
I20240501 14:00:23.537586 23251 raft_consensus.cc:482] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.537796 23251 raft_consensus.cc:504] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:23.538430 23251 leader_election.cc:288] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:23.539141 22412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28e7ed3a23243ee98327f60b2ebdbd4" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" is_pre_election: true
I20240501 14:00:23.539402 22412 raft_consensus.cc:2454] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 0.
I20240501 14:00:23.539908 22479 leader_election.cc:302] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:23.540271 23251 raft_consensus.cc:2781] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.540414 23251 raft_consensus.cc:482] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.540500 23251 raft_consensus.cc:3037] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.543423 23251 raft_consensus.cc:504] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:23.544019 23251 leader_election.cc:288] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Requested vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:23.544663 22412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28e7ed3a23243ee98327f60b2ebdbd4" candidate_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
I20240501 14:00:23.544869 22412 raft_consensus.cc:3037] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.546905 22412 raft_consensus.cc:2454] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1653dc15114e7c873f9cc1a9a3c0a5 in term 1.
I20240501 14:00:23.547418 22479 leader_election.cc:302] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3b1653dc15114e7c873f9cc1a9a3c0a5, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:00:23.547788 23251 raft_consensus.cc:2781] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.548200 23251 raft_consensus.cc:686] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Running, Role: LEADER
I20240501 14:00:23.548786 23251 consensus_queue.cc:237] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } }
I20240501 14:00:23.551805 22140 catalog_manager.cc:5484] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } }
I20240501 14:00:23.753357 23399 consensus_queue.cc:1035] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:23.767756 23357 consensus_queue.cc:1035] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:23.941463 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:23.941890 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:23.950006 23457 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:00:24.045997 23251 consensus_queue.cc:1035] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 251.773 ms
time per row: 0.125887 ms
I20240501 14:00:24.353183 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:45725
--local_ip_for_outbound_sockets=127.16.36.132
--tserver_master_addrs=127.16.36.190:35677
--webserver_port=37233
--webserver_interface=127.16.36.132
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:00:24.708930 23510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:24.709282 23510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:24.709437 23510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:24.717239 23510 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:24.717424 23510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:24.717589 23510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 14:00:24.727893 23510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:45725
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=37233
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:24.729308 23510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:24.731163 23510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:24.744879 23515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:24.745430 23518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:24.746054 23516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:24.746783 23510 server_base.cc:1034] running on GCE node
I20240501 14:00:24.747471 23510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:24.748566 23510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:24.749819 23510 hybrid_clock.cc:648] HybridClock initialized: now 1714572024749771 us; error 43 us; skew 500 ppm
I20240501 14:00:24.750169 23510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:24.752753 23510 webserver.cc:460] Webserver started at http://127.16.36.132:37233/ using document root <none> and password file <none>
I20240501 14:00:24.753602 23510 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:24.753749 23510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:24.759742 23510 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:24.763288 23524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:24.764786 23510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:24.764945 23510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "90f980f10dfd441289a8ad090e23e200"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:24.765826 23510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:24.787742 23510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:24.788620 23510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:24.788901 23510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:24.789829 23510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:24.791422 23510 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:24.791512 23510 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:24.791649 23510 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:24.791716 23510 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:24.824853 23510 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:45725
I20240501 14:00:24.824899 23636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:45725 every 8 connection(s)
I20240501 14:00:24.826380 23510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:00:24.832901 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 23510
I20240501 14:00:24.833586 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:39481
--local_ip_for_outbound_sockets=127.16.36.133
--tserver_master_addrs=127.16.36.190:35677
--webserver_port=33409
--webserver_interface=127.16.36.133
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:24.839995 23637 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:24.840384 23637 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:24.841395 23637 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:24.843364 22139 ts_manager.cc:194] Re-registered known tserver with Master: 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725)
I20240501 14:00:24.844278 22139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:42661
W20240501 14:00:25.183769 23641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:25.184134 23641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:25.184289 23641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:25.193600 23641 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:25.193750 23641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:25.193920 23641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 14:00:25.203831 23641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:39481
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=33409
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:25.205260 23641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:25.207201 23641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:25.220273 23647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:25.220955 23648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:25.221621 23650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:25.222198 23641 server_base.cc:1034] running on GCE node
I20240501 14:00:25.222967 23641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:25.224232 23641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:25.225450 23641 hybrid_clock.cc:648] HybridClock initialized: now 1714572025225387 us; error 55 us; skew 500 ppm
I20240501 14:00:25.225841 23641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:25.228516 23641 webserver.cc:460] Webserver started at http://127.16.36.133:33409/ using document root <none> and password file <none>
I20240501 14:00:25.229261 23641 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:25.229375 23641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:25.234902 23641 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:25.237987 23656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:25.239727 23641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:25.239923 23641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5e1c3e3144384f90b47255623d63575b"
format_stamp: "Formatted at 2024-05-01 14:00:12 on dist-test-slave-9q9t"
I20240501 14:00:25.240878 23641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:25.283967 23641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:25.285048 23641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:25.285370 23641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:25.286546 23641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:25.288703 23663 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 14:00:25.294663 23641 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:00:25.294793 23641 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20240501 14:00:25.294951 23641 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:00:25.298002 23641 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:00:25.298117 23641 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240501 14:00:25.298578 23663 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Bootstrap starting.
I20240501 14:00:25.319962 23663 log.cc:824] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:25.331732 23663 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:25.333016 23663 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Bootstrap complete.
I20240501 14:00:25.335005 23663 ts_tablet_manager.cc:1367] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.015s
I20240501 14:00:25.339839 23641 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:39481
I20240501 14:00:25.339896 23770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:39481 every 8 connection(s)
I20240501 14:00:25.341725 23641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:00:25.345796 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 23641
I20240501 14:00:25.346612 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.134:35555
--local_ip_for_outbound_sockets=127.16.36.134
--tserver_master_addrs=127.16.36.190:35677
--webserver_port=39539
--webserver_interface=127.16.36.134
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:25.347678 23663 raft_consensus.cc:348] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:25.348955 23663 raft_consensus.cc:729] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.349854 23663 consensus_queue.cc:260] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:25.353606 23663 ts_tablet_manager.cc:1398] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20240501 14:00:25.357156 23771 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:25.357465 23771 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:25.358193 23771 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:25.360710 22139 ts_manager.cc:194] Re-registered known tserver with Master: 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481)
I20240501 14:00:25.361939 22139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:42535
I20240501 14:00:25.365447 23771 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
W20240501 14:00:25.593638 23772 tablet.cc:2369] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:00:25.697530 23776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:25.697853 23776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:25.697953 23776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:25.708192 23776 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:25.708331 23776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:25.708513 23776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.134
I20240501 14:00:25.718107 23776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.134:35555
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.134
--webserver_port=39539
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.134
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:25.719535 23776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:25.721275 23776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:25.734202 23781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:25.734756 23784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:25.735321 23782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:25.736075 23776 server_base.cc:1034] running on GCE node
I20240501 14:00:25.736853 23776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:25.737926 23776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:25.739132 23776 hybrid_clock.cc:648] HybridClock initialized: now 1714572025739075 us; error 55 us; skew 500 ppm
I20240501 14:00:25.739477 23776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:25.741922 23776 webserver.cc:460] Webserver started at http://127.16.36.134:39539/ using document root <none> and password file <none>
I20240501 14:00:25.742730 23776 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:25.742862 23776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:25.748250 23776 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 14:00:25.751070 23790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:25.752440 23776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:00:25.752612 23776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "dedba8a189394baca3188f2a5eddbcd4"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:25.753453 23776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:25.779009 23776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:25.779870 23776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:25.780154 23776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:25.781165 23776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:25.782855 23776 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:25.782954 23776 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:25.783087 23776 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:25.783149 23776 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:25.816682 23776 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.134:35555
I20240501 14:00:25.816736 23902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.134:35555 every 8 connection(s)
I20240501 14:00:25.818217 23776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:00:25.820268 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 23776
I20240501 14:00:25.820878 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.135:41675
--local_ip_for_outbound_sockets=127.16.36.135
--tserver_master_addrs=127.16.36.190:35677
--webserver_port=45885
--webserver_interface=127.16.36.135
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:00:25.833503 23903 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:25.833861 23903 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:25.834851 23903 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:25.836580 22139 ts_manager.cc:194] Re-registered known tserver with Master: dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555)
I20240501 14:00:25.837924 22139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.134:56935
I20240501 14:00:25.847690 23637 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
W20240501 14:00:26.174492 23906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:26.174955 23906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:26.175108 23906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:26.183255 23906 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:00:26.183398 23906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:26.183565 23906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.135
I20240501 14:00:26.193665 23906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.135:41675
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.135
--webserver_port=45885
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.135
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:26.195523 23906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:26.197499 23906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:26.210867 23911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:26.211412 23914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:26.212183 23912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:26.212800 23906 server_base.cc:1034] running on GCE node
I20240501 14:00:26.213465 23906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:26.214501 23906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:26.215701 23906 hybrid_clock.cc:648] HybridClock initialized: now 1714572026215637 us; error 59 us; skew 500 ppm
I20240501 14:00:26.216037 23906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:26.218600 23906 webserver.cc:460] Webserver started at http://127.16.36.135:45885/ using document root <none> and password file <none>
I20240501 14:00:26.219421 23906 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:26.219547 23906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:26.225171 23906 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20240501 14:00:26.228304 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:26.229629 23906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:26.229815 23906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "bcf471e1f6eb42b8a9ae457e6239b89f"
format_stamp: "Formatted at 2024-05-01 14:00:13 on dist-test-slave-9q9t"
I20240501 14:00:26.230697 23906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:26.262384 23906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:26.263280 23906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:26.263566 23906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:26.264583 23906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:26.266494 23927 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 14:00:26.272161 23906 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:00:26.272277 23906 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 14:00:26.272390 23906 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:00:26.275528 23906 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:00:26.275635 23906 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:26.276098 23927 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap starting.
I20240501 14:00:26.296665 23927 log.cc:824] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:26.300283 23927 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:26.301100 23927 tablet_bootstrap.cc:492] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap complete.
I20240501 14:00:26.303185 23927 ts_tablet_manager.cc:1367] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.011s
I20240501 14:00:26.314747 23906 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.135:41675
I20240501 14:00:26.314791 24034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.135:41675 every 8 connection(s)
I20240501 14:00:26.316783 23906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:00:26.317937 23927 raft_consensus.cc:348] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:26.319036 23927 raft_consensus.cc:729] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf471e1f6eb42b8a9ae457e6239b89f, State: Initialized, Role: FOLLOWER
I20240501 14:00:26.319846 23927 consensus_queue.cc:260] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:26.321717 23927 ts_tablet_manager.cc:1398] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20240501 14:00:26.325539 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 23906
I20240501 14:00:26.325852 16530 test_util.cc:268] Using random seed: -393073875
I20240501 14:00:26.330293 24035 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:00:26.330658 24035 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:26.331560 24035 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:00:26.334512 22139 ts_manager.cc:194] Re-registered known tserver with Master: bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675)
I20240501 14:00:26.336136 22139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.135:33023
I20240501 14:00:26.339064 24035 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
I20240501 14:00:26.344208 16530 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 14:00:26.344746 16530 test_util.cc:268] Using random seed: -393054981
I20240501 14:00:26.365883 16530 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 14:00:26.366379 16530 test_util.cc:268] Using random seed: -393033356
I20240501 14:00:26.383813 16530 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 14:00:26.384435 16530 test_util.cc:268] Using random seed: -393015314
I20240501 14:00:26.443289 16530 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 14:00:26.459424 16530 test_util.cc:268] Using random seed: -392940339
I20240501 14:00:26.503537 16530 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240501 14:00:26.569730 24036 tablet.cc:2369] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:26.718523 24130 raft_consensus.cc:482] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:26.718932 24130 raft_consensus.cc:504] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:26.721082 24130 leader_election.cc:288] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675)
I20240501 14:00:26.758446 23990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce58c7fa6c734d3fb453d132540f933c" candidate_uuid: "5e1c3e3144384f90b47255623d63575b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" is_pre_election: true
I20240501 14:00:26.759126 23990 raft_consensus.cc:2454] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c3e3144384f90b47255623d63575b in term 1.
I20240501 14:00:26.760740 23660 leader_election.cc:302] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5e1c3e3144384f90b47255623d63575b, bcf471e1f6eb42b8a9ae457e6239b89f; no voters:
I20240501 14:00:26.761265 24130 raft_consensus.cc:2781] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:00:26.761430 24130 raft_consensus.cc:482] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:26.761526 24130 raft_consensus.cc:3037] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:26.766886 24130 raft_consensus.cc:504] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:26.767930 24130 leader_election.cc:288] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 2 election: Requested vote from peers bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675)
I20240501 14:00:26.768965 23990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce58c7fa6c734d3fb453d132540f933c" candidate_uuid: "5e1c3e3144384f90b47255623d63575b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f"
I20240501 14:00:26.769194 23990 raft_consensus.cc:3037] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:26.774497 23990 raft_consensus.cc:2454] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c3e3144384f90b47255623d63575b in term 2.
I20240501 14:00:26.776875 23660 leader_election.cc:302] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5e1c3e3144384f90b47255623d63575b, bcf471e1f6eb42b8a9ae457e6239b89f; no voters:
I20240501 14:00:26.777520 24130 raft_consensus.cc:2781] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:26.821166 24130 raft_consensus.cc:686] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Running, Role: LEADER
I20240501 14:00:26.821853 24130 consensus_queue.cc:237] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:26.841073 23903 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
I20240501 14:00:26.848419 22139 catalog_manager.cc:5484] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "5e1c3e3144384f90b47255623d63575b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:00:27.416110 23990 raft_consensus.cc:1261] T ce58c7fa6c734d3fb453d132540f933c P bcf471e1f6eb42b8a9ae457e6239b89f [term 2 FOLLOWER]: Refusing update from remote peer 5e1c3e3144384f90b47255623d63575b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:00:27.417106 24159 consensus_queue.cc:1035] T ce58c7fa6c734d3fb453d132540f933c P 5e1c3e3144384f90b47255623d63575b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:27.838543 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:27.875460 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:27.882117 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:27.893013 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:27.905779 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:27.944226 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:28.016393 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:00:28.199734 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:33963, user_credentials={real_user=slave}} blocked reactor thread for 58960.1us
I20240501 14:00:28.987171 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:28.992857 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:28.994766 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:28.999511 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:29.054219 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:29.072110 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:29.085028 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:29.524583 22543 consensus_queue.cc:237] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:29.535898 22281 raft_consensus.cc:1261] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240501 14:00:29.539774 24065 consensus_queue.cc:1035] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
W20240501 14:00:29.542333 22479 consensus_peers.cc:485] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555): Couldn't send request to peer dedba8a189394baca3188f2a5eddbcd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c40fd7b8bfd04d8ca67a9acf40f74445. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:29.547271 24205 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:29.547974 22281 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:29.556922 22139 catalog_manager.cc:5484] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index -1 to 60, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:29.840260 22412 consensus_queue.cc:237] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:29.851465 22540 raft_consensus.cc:1261] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240501 14:00:29.856916 24129 consensus_queue.cc:1035] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240501 14:00:29.887552 22541 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:29.886376 23642 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:29.899585 22140 catalog_manager.cc:5484] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index -1 to 61, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
W20240501 14:00:29.914806 22346 consensus_peers.cc:485] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c -> Peer 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725): Couldn't send request to peer 90f980f10dfd441289a8ad090e23e200. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c09f45be8c064ecdbdc78d5ce8dbe8bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:30.107457 24315 ts_tablet_manager.cc:902] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Initiating tablet copy from peer 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:30.129683 22540 consensus_queue.cc:237] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:30.135838 22413 raft_consensus.cc:1261] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 14:00:30.136586 24144 consensus_queue.cc:1035] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 14:00:30.139936 24144 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:00:30.146754 22139 catalog_manager.cc:5484] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index -1 to 79, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:00:30.152169 22412 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
W20240501 14:00:30.165091 22476 consensus_peers.cc:485] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481): Couldn't send request to peer 5e1c3e3144384f90b47255623d63575b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b28e7ed3a23243ee98327f60b2ebdbd4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:30.236202 24325 ts_tablet_manager.cc:902] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:30.254657 24315 tablet_copy_client.cc:308] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:38305
I20240501 14:00:30.310727 22563 tablet_copy_service.cc:140] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Received BeginTabletCopySession request for tablet c40fd7b8bfd04d8ca67a9acf40f74445 from peer dedba8a189394baca3188f2a5eddbcd4 ({username='slave'} at 127.16.36.134:41405)
I20240501 14:00:30.311142 22563 tablet_copy_service.cc:161] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Beginning new tablet copy session on tablet c40fd7b8bfd04d8ca67a9acf40f74445 from peer dedba8a189394baca3188f2a5eddbcd4 at {username='slave'} at 127.16.36.134:41405: session id = dedba8a189394baca3188f2a5eddbcd4-c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:00:30.330749 22563 tablet_copy_source_session.cc:205] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:30.342468 24315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40fd7b8bfd04d8ca67a9acf40f74445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.391770 24315 tablet_copy_client.cc:783] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:30.392778 24315 tablet_copy_client.cc:647] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:30.396649 24315 tablet_copy_client.cc:523] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:30.402408 24315 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap starting.
I20240501 14:00:30.412290 24325 tablet_copy_client.cc:308] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:00:30.456697 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf from peer 90f980f10dfd441289a8ad090e23e200 ({username='slave'} at 127.16.36.132:35525)
I20240501 14:00:30.457000 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet c09f45be8c064ecdbdc78d5ce8dbe8bf from peer 90f980f10dfd441289a8ad090e23e200 at {username='slave'} at 127.16.36.132:35525: session id = 90f980f10dfd441289a8ad090e23e200-c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:00:30.463169 22433 tablet_copy_source_session.cc:205] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:30.480448 24315 log.cc:824] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:30.510152 24325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.570873 24325 tablet_copy_client.cc:783] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:30.583901 24325 tablet_copy_client.cc:647] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:30.591305 24325 tablet_copy_client.cc:523] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:30.618950 24325 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: Bootstrap starting.
I20240501 14:00:30.720624 24346 ts_tablet_manager.cc:902] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: Initiating tablet copy from peer 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:30.768013 24325 log.cc:824] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:30.776353 24315 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=72 ignored=0} inserts{seen=703 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:30.777292 24346 tablet_copy_client.cc:308] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:38305
I20240501 14:00:30.793532 24315 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap complete.
I20240501 14:00:30.794425 24315 ts_tablet_manager.cc:1367] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Time spent bootstrapping tablet: real 0.393s user 0.158s sys 0.016s
I20240501 14:00:30.812948 22563 tablet_copy_service.cc:140] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Received BeginTabletCopySession request for tablet b28e7ed3a23243ee98327f60b2ebdbd4 from peer 5e1c3e3144384f90b47255623d63575b ({username='slave'} at 127.16.36.133:51451)
I20240501 14:00:30.813158 22563 tablet_copy_service.cc:161] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Beginning new tablet copy session on tablet b28e7ed3a23243ee98327f60b2ebdbd4 from peer 5e1c3e3144384f90b47255623d63575b at {username='slave'} at 127.16.36.133:51451: session id = 5e1c3e3144384f90b47255623d63575b-b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:00:30.816022 22563 tablet_copy_source_session.cc:205] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:30.824275 24346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28e7ed3a23243ee98327f60b2ebdbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.858753 24346 tablet_copy_client.cc:783] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:30.860101 24346 tablet_copy_client.cc:647] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:30.863001 24346 tablet_copy_client.cc:523] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:30.870342 24346 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: Bootstrap starting.
I20240501 14:00:30.872521 24315 raft_consensus.cc:348] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:30.880533 24315 raft_consensus.cc:729] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedba8a189394baca3188f2a5eddbcd4, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.881757 24315 consensus_queue.cc:260] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:30.904317 22412 consensus_queue.cc:237] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:30.921918 24315 ts_tablet_manager.cc:1398] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Time spent starting tablet: real 0.125s user 0.006s sys 0.005s
I20240501 14:00:30.923890 22563 tablet_copy_service.cc:342] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Request end of tablet copy session dedba8a189394baca3188f2a5eddbcd4-c40fd7b8bfd04d8ca67a9acf40f74445 received from {username='slave'} at 127.16.36.134:41405
I20240501 14:00:30.924117 22563 tablet_copy_service.cc:433] P 3b1653dc15114e7c873f9cc1a9a3c0a5: ending tablet copy session dedba8a189394baca3188f2a5eddbcd4-c40fd7b8bfd04d8ca67a9acf40f74445 on tablet c40fd7b8bfd04d8ca67a9acf40f74445 with peer dedba8a189394baca3188f2a5eddbcd4
I20240501 14:00:30.930589 22280 raft_consensus.cc:1261] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240501 14:00:30.942687 24158 consensus_queue.cc:1035] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 14:00:30.957480 24298 raft_consensus.cc:2932] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:30.967268 22139 catalog_manager.cc:5484] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index -1 to 64, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:30.989024 22280 raft_consensus.cc:2932] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:31.055533 24325 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=77 ignored=0} inserts{seen=749 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:31.056236 24325 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: Bootstrap complete.
I20240501 14:00:31.056803 24325 ts_tablet_manager.cc:1367] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: Time spent bootstrapping tablet: real 0.439s user 0.169s sys 0.010s
I20240501 14:00:31.064028 24325 raft_consensus.cc:348] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:31.064543 24325 raft_consensus.cc:729] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f980f10dfd441289a8ad090e23e200, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.065341 24325 consensus_queue.cc:260] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:31.071302 24325 ts_tablet_manager.cc:1398] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20240501 14:00:31.075134 22349 consensus_peers.cc:485] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c -> Peer bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675): Couldn't send request to peer bcf471e1f6eb42b8a9ae457e6239b89f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 161faaa672ce4a0e80495369f1060558. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:31.079274 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session 90f980f10dfd441289a8ad090e23e200-c09f45be8c064ecdbdc78d5ce8dbe8bf received from {username='slave'} at 127.16.36.132:35525
I20240501 14:00:31.079493 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session 90f980f10dfd441289a8ad090e23e200-c09f45be8c064ecdbdc78d5ce8dbe8bf on tablet c09f45be8c064ecdbdc78d5ce8dbe8bf with peer 90f980f10dfd441289a8ad090e23e200
I20240501 14:00:31.245764 24346 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=764 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:31.246537 24346 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: Bootstrap complete.
I20240501 14:00:31.247053 24346 ts_tablet_manager.cc:1367] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: Time spent bootstrapping tablet: real 0.377s user 0.180s sys 0.016s
I20240501 14:00:31.249107 24346 raft_consensus.cc:348] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:31.249470 24346 raft_consensus.cc:729] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.249790 24346 consensus_queue.cc:260] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 1.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:31.251232 24346 ts_tablet_manager.cc:1398] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:31.253249 22563 tablet_copy_service.cc:342] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Request end of tablet copy session 5e1c3e3144384f90b47255623d63575b-b28e7ed3a23243ee98327f60b2ebdbd4 received from {username='slave'} at 127.16.36.133:51451
I20240501 14:00:31.253403 22563 tablet_copy_service.cc:433] P 3b1653dc15114e7c873f9cc1a9a3c0a5: ending tablet copy session 5e1c3e3144384f90b47255623d63575b-b28e7ed3a23243ee98327f60b2ebdbd4 on tablet b28e7ed3a23243ee98327f60b2ebdbd4 with peer 5e1c3e3144384f90b47255623d63575b
I20240501 14:00:31.257871 23858 raft_consensus.cc:1203] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.90] Dedup: 1.72->[1.73-1.90]
I20240501 14:00:31.305060 23592 raft_consensus.cc:1203] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.86] Dedup: 1.77->[1.78-1.86]
I20240501 14:00:31.481673 24400 ts_tablet_manager.cc:902] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:31.483500 22413 consensus_queue.cc:237] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:31.501665 22542 raft_consensus.cc:1261] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240501 14:00:31.503103 24293 consensus_queue.cc:1035] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 86, Time since last communication: 0.000s
I20240501 14:00:31.517352 24361 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:31.519626 24400 tablet_copy_client.cc:308] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:00:31.524808 22541 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
W20240501 14:00:31.526610 22349 consensus_peers.cc:485] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c -> Peer dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555): Couldn't send request to peer dedba8a189394baca3188f2a5eddbcd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 999759caca8344f2b2ae0cff5e204425. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:31.539999 22140 catalog_manager.cc:5484] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index -1 to 88, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:31.585650 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet 161faaa672ce4a0e80495369f1060558 from peer bcf471e1f6eb42b8a9ae457e6239b89f ({username='slave'} at 127.16.36.135:54763)
I20240501 14:00:31.585914 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet 161faaa672ce4a0e80495369f1060558 from peer bcf471e1f6eb42b8a9ae457e6239b89f at {username='slave'} at 127.16.36.135:54763: session id = bcf471e1f6eb42b8a9ae457e6239b89f-161faaa672ce4a0e80495369f1060558
I20240501 14:00:31.588871 22433 tablet_copy_source_session.cc:205] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:31.595597 24400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161faaa672ce4a0e80495369f1060558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:31.604775 24400 tablet_copy_client.cc:783] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:31.605166 24400 tablet_copy_client.cc:647] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:31.608311 24400 tablet_copy_client.cc:523] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:31.614028 24400 tablet_bootstrap.cc:492] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap starting.
I20240501 14:00:31.832794 23726 raft_consensus.cc:1203] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.78->[1.79-1.111] Dedup: 1.92->[1.93-1.111]
I20240501 14:00:31.925838 24432 ts_tablet_manager.cc:902] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:31.930241 24400 tablet_bootstrap.cc:492] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=71 ignored=0} inserts{seen=756 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:31.930928 24400 tablet_bootstrap.cc:492] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap complete.
I20240501 14:00:31.931258 24400 ts_tablet_manager.cc:1367] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent bootstrapping tablet: real 0.317s user 0.144s sys 0.022s
I20240501 14:00:31.934490 24400 raft_consensus.cc:348] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:31.959510 24400 raft_consensus.cc:729] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf471e1f6eb42b8a9ae457e6239b89f, State: Initialized, Role: FOLLOWER
I20240501 14:00:31.960564 24400 consensus_queue.cc:260] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:31.997188 24432 tablet_copy_client.cc:308] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:00:32.006428 24400 ts_tablet_manager.cc:1398] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent starting tablet: real 0.075s user 0.007s sys 0.022s
I20240501 14:00:32.024199 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session bcf471e1f6eb42b8a9ae457e6239b89f-161faaa672ce4a0e80495369f1060558 received from {username='slave'} at 127.16.36.135:54763
I20240501 14:00:32.024376 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session bcf471e1f6eb42b8a9ae457e6239b89f-161faaa672ce4a0e80495369f1060558 on tablet 161faaa672ce4a0e80495369f1060558 with peer bcf471e1f6eb42b8a9ae457e6239b89f
I20240501 14:00:32.028838 22541 consensus_queue.cc:237] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:32.030181 23990 raft_consensus.cc:1203] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.75] Dedup: 1.72->[1.73-1.75]
I20240501 14:00:32.043125 22413 raft_consensus.cc:1261] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 14:00:32.044134 24380 consensus_queue.cc:1035] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240501 14:00:32.049082 24380 raft_consensus.cc:2932] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:32.055298 22411 raft_consensus.cc:2932] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:32.069590 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet 999759caca8344f2b2ae0cff5e204425 from peer dedba8a189394baca3188f2a5eddbcd4 ({username='slave'} at 127.16.36.134:57731)
I20240501 14:00:32.069808 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet 999759caca8344f2b2ae0cff5e204425 from peer dedba8a189394baca3188f2a5eddbcd4 at {username='slave'} at 127.16.36.134:57731: session id = dedba8a189394baca3188f2a5eddbcd4-999759caca8344f2b2ae0cff5e204425
I20240501 14:00:32.072075 22140 catalog_manager.cc:5484] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index -1 to 98, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:32.072816 22433 tablet_copy_source_session.cc:205] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:32.079943 24432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999759caca8344f2b2ae0cff5e204425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.089725 24432 tablet_copy_client.cc:783] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:32.090034 24432 tablet_copy_client.cc:647] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:32.101164 24432 tablet_copy_client.cc:523] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:32.106389 24432 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap starting.
W20240501 14:00:32.146661 22476 consensus_peers.cc:485] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725): Couldn't send request to peer 90f980f10dfd441289a8ad090e23e200. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74e03b7d269c40ff9fbf6457611d849b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:32.361361 22411 consensus_queue.cc:237] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:32.402348 22281 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240501 14:00:32.406865 24361 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 14:00:32.411293 24361 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:00:32.423323 24478 ts_tablet_manager.cc:902] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: Initiating tablet copy from peer 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:32.426147 22139 catalog_manager.cc:5484] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index -1 to 77, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:00:32.434008 22281 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
W20240501 14:00:32.443231 22346 consensus_peers.cc:485] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c -> Peer 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481): Couldn't send request to peer 5e1c3e3144384f90b47255623d63575b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc19025daec44883afec64b3d12c7c4e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:32.509810 24432 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:32.510454 24432 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap complete.
I20240501 14:00:32.511381 24432 ts_tablet_manager.cc:1367] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: Time spent bootstrapping tablet: real 0.405s user 0.161s sys 0.061s
I20240501 14:00:32.512719 24432 raft_consensus.cc:348] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:32.513101 24432 raft_consensus.cc:729] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedba8a189394baca3188f2a5eddbcd4, State: Initialized, Role: FOLLOWER
I20240501 14:00:32.513413 24432 consensus_queue.cc:260] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:32.523334 24432 ts_tablet_manager.cc:1398] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4: Time spent starting tablet: real 0.012s user 0.002s sys 0.008s
I20240501 14:00:32.535785 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session dedba8a189394baca3188f2a5eddbcd4-999759caca8344f2b2ae0cff5e204425 received from {username='slave'} at 127.16.36.134:57731
I20240501 14:00:32.535966 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session dedba8a189394baca3188f2a5eddbcd4-999759caca8344f2b2ae0cff5e204425 on tablet 999759caca8344f2b2ae0cff5e204425 with peer dedba8a189394baca3188f2a5eddbcd4
I20240501 14:00:32.542919 24478 tablet_copy_client.cc:308] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:38305
I20240501 14:00:32.544981 23857 raft_consensus.cc:1203] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.87->[1.88-1.96] Dedup: 1.91->[1.92-1.96]
I20240501 14:00:32.790424 24495 ts_tablet_manager.cc:902] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:32.832216 24495 tablet_copy_client.cc:308] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:00:32.845985 22563 tablet_copy_service.cc:140] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Received BeginTabletCopySession request for tablet 74e03b7d269c40ff9fbf6457611d849b from peer 90f980f10dfd441289a8ad090e23e200 ({username='slave'} at 127.16.36.132:37671)
I20240501 14:00:32.846210 22563 tablet_copy_service.cc:161] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Beginning new tablet copy session on tablet 74e03b7d269c40ff9fbf6457611d849b from peer 90f980f10dfd441289a8ad090e23e200 at {username='slave'} at 127.16.36.132:37671: session id = 90f980f10dfd441289a8ad090e23e200-74e03b7d269c40ff9fbf6457611d849b
I20240501 14:00:32.848861 22563 tablet_copy_source_session.cc:205] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:32.850313 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e03b7d269c40ff9fbf6457611d849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.856818 24478 tablet_copy_client.cc:783] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:32.857107 24478 tablet_copy_client.cc:647] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:32.863232 24478 tablet_copy_client.cc:523] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:32.880308 24478 tablet_bootstrap.cc:492] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: Bootstrap starting.
I20240501 14:00:32.914264 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet dc19025daec44883afec64b3d12c7c4e from peer 5e1c3e3144384f90b47255623d63575b ({username='slave'} at 127.16.36.133:45659)
I20240501 14:00:32.914531 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet dc19025daec44883afec64b3d12c7c4e from peer 5e1c3e3144384f90b47255623d63575b at {username='slave'} at 127.16.36.133:45659: session id = 5e1c3e3144384f90b47255623d63575b-dc19025daec44883afec64b3d12c7c4e
I20240501 14:00:32.943611 22433 tablet_copy_source_session.cc:205] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:32.945245 24495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19025daec44883afec64b3d12c7c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:32.953043 24495 tablet_copy_client.cc:783] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:32.953344 24495 tablet_copy_client.cc:647] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:32.963353 24495 tablet_copy_client.cc:523] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:32.970448 24495 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: Bootstrap starting.
I20240501 14:00:33.081867 22541 consensus_queue.cc:237] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:33.104409 22281 raft_consensus.cc:1261] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240501 14:00:33.107528 24380 consensus_queue.cc:1035] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240501 14:00:33.131807 24334 raft_consensus.cc:2932] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index -1 to 100, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
W20240501 14:00:33.140189 22479 consensus_peers.cc:485] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675): Couldn't send request to peer bcf471e1f6eb42b8a9ae457e6239b89f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd5f6e034ac14cd2864d894fe5b00fd9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:33.141463 24478 tablet_bootstrap.cc:492] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=108 ignored=0} inserts{seen=780 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:33.141966 24478 tablet_bootstrap.cc:492] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: Bootstrap complete.
I20240501 14:00:33.142284 24478 ts_tablet_manager.cc:1367] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: Time spent bootstrapping tablet: real 0.263s user 0.176s sys 0.009s
I20240501 14:00:33.143314 24478 raft_consensus.cc:348] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:33.143556 24478 raft_consensus.cc:729] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f980f10dfd441289a8ad090e23e200, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.143790 24478 consensus_queue.cc:260] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:33.150139 22139 catalog_manager.cc:5484] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index -1 to 100, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:33.154078 22281 raft_consensus.cc:2932] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index -1 to 100, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:33.168968 24478 ts_tablet_manager.cc:1398] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200: Time spent starting tablet: real 0.027s user 0.012s sys 0.002s
I20240501 14:00:33.171195 22563 tablet_copy_service.cc:342] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Request end of tablet copy session 90f980f10dfd441289a8ad090e23e200-74e03b7d269c40ff9fbf6457611d849b received from {username='slave'} at 127.16.36.132:37671
I20240501 14:00:33.171391 22563 tablet_copy_service.cc:433] P 3b1653dc15114e7c873f9cc1a9a3c0a5: ending tablet copy session 90f980f10dfd441289a8ad090e23e200-74e03b7d269c40ff9fbf6457611d849b on tablet 74e03b7d269c40ff9fbf6457611d849b with peer 90f980f10dfd441289a8ad090e23e200
I20240501 14:00:33.255084 23592 raft_consensus.cc:1203] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.97->[1.98-1.117] Dedup: 1.108->[1.109-1.117]
I20240501 14:00:33.370157 24495 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=80 ignored=0} inserts{seen=765 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:33.379451 24495 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: Bootstrap complete.
I20240501 14:00:33.380224 24495 ts_tablet_manager.cc:1367] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: Time spent bootstrapping tablet: real 0.410s user 0.137s sys 0.026s
I20240501 14:00:33.381703 24495 raft_consensus.cc:348] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:33.390619 24495 raft_consensus.cc:729] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.392155 24495 consensus_queue.cc:260] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:33.393507 24495 ts_tablet_manager.cc:1398] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:00:33.402415 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session 5e1c3e3144384f90b47255623d63575b-dc19025daec44883afec64b3d12c7c4e received from {username='slave'} at 127.16.36.133:45659
I20240501 14:00:33.402588 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session 5e1c3e3144384f90b47255623d63575b-dc19025daec44883afec64b3d12c7c4e on tablet dc19025daec44883afec64b3d12c7c4e with peer 5e1c3e3144384f90b47255623d63575b
I20240501 14:00:33.488441 24540 ts_tablet_manager.cc:902] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: Initiating tablet copy from peer 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:00:33.517691 24540 tablet_copy_client.cc:308] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:38305
I20240501 14:00:33.545720 22563 tablet_copy_service.cc:140] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Received BeginTabletCopySession request for tablet dd5f6e034ac14cd2864d894fe5b00fd9 from peer bcf471e1f6eb42b8a9ae457e6239b89f ({username='slave'} at 127.16.36.135:60485)
I20240501 14:00:33.545943 22563 tablet_copy_service.cc:161] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Beginning new tablet copy session on tablet dd5f6e034ac14cd2864d894fe5b00fd9 from peer bcf471e1f6eb42b8a9ae457e6239b89f at {username='slave'} at 127.16.36.135:60485: session id = bcf471e1f6eb42b8a9ae457e6239b89f-dd5f6e034ac14cd2864d894fe5b00fd9
I20240501 14:00:33.548890 22563 tablet_copy_source_session.cc:205] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:33.550494 24540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5f6e034ac14cd2864d894fe5b00fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.559491 24540 tablet_copy_client.cc:783] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:33.559939 24540 tablet_copy_client.cc:647] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:33.562403 24540 tablet_copy_client.cc:523] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:33.566280 24540 tablet_bootstrap.cc:492] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap starting.
I20240501 14:00:33.696346 22280 consensus_queue.cc:237] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.89, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:33.726424 22413 raft_consensus.cc:1261] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 90. (index mismatch)
I20240501 14:00:33.730790 24499 consensus_queue.cc:1035] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 88, Time since last communication: 0.000s
W20240501 14:00:33.743357 22218 consensus_peers.cc:485] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 -> Peer dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555): Couldn't send request to peer dedba8a189394baca3188f2a5eddbcd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4716a6313a541b981e5d27b01657956. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:33.754514 24113 raft_consensus.cc:2932] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Committing config change with OpId 1.90: config changed from index -1 to 90, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:33.758682 22412 raft_consensus.cc:2932] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Committing config change with OpId 1.90: config changed from index -1 to 90, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:33.765020 22139 catalog_manager.cc:5484] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index -1 to 90, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:33.786484 23726 raft_consensus.cc:1203] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.76->[1.77-1.85] Dedup: 1.81->[1.82-1.85]
I20240501 14:00:33.967535 24540 tablet_bootstrap.cc:492] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap replayed 1/1 log segments. Stats: ops{read=105 overwritten=0 applied=105 ignored=0} inserts{seen=736 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:33.968156 24540 tablet_bootstrap.cc:492] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap complete.
I20240501 14:00:33.968537 24540 ts_tablet_manager.cc:1367] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent bootstrapping tablet: real 0.402s user 0.171s sys 0.013s
I20240501 14:00:33.969435 24540 raft_consensus.cc:348] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:33.969712 24540 raft_consensus.cc:729] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf471e1f6eb42b8a9ae457e6239b89f, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.970010 24540 consensus_queue.cc:260] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:34.002929 24540 ts_tablet_manager.cc:1398] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent starting tablet: real 0.034s user 0.014s sys 0.019s
I20240501 14:00:34.003968 22563 tablet_copy_service.cc:342] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Request end of tablet copy session bcf471e1f6eb42b8a9ae457e6239b89f-dd5f6e034ac14cd2864d894fe5b00fd9 received from {username='slave'} at 127.16.36.135:60485
I20240501 14:00:34.004115 22563 tablet_copy_service.cc:433] P 3b1653dc15114e7c873f9cc1a9a3c0a5: ending tablet copy session bcf471e1f6eb42b8a9ae457e6239b89f-dd5f6e034ac14cd2864d894fe5b00fd9 on tablet dd5f6e034ac14cd2864d894fe5b00fd9 with peer bcf471e1f6eb42b8a9ae457e6239b89f
I20240501 14:00:34.073992 23990 raft_consensus.cc:1203] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.99->[1.100-1.110] Dedup: 1.105->[1.106-1.110]
I20240501 14:00:34.261570 24586 ts_tablet_manager.cc:902] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: Initiating tablet copy from peer 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:34.311779 22280 consensus_queue.cc:237] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:34.329044 22542 raft_consensus.cc:1261] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 14:00:34.360759 24586 tablet_copy_client.cc:308] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:33963
I20240501 14:00:34.370739 24561 consensus_queue.cc:1035] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.014s
W20240501 14:00:34.376093 22215 consensus_peers.cc:485] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 -> Peer 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725): Couldn't send request to peer 90f980f10dfd441289a8ad090e23e200. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6ac874a7f084076bf03ed4167f967e0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:34.380223 24499 raft_consensus.cc:2932] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:34.392359 22302 tablet_copy_service.cc:140] P 474d063159a143fcba0416fa01328bc0: Received BeginTabletCopySession request for tablet d4716a6313a541b981e5d27b01657956 from peer dedba8a189394baca3188f2a5eddbcd4 ({username='slave'} at 127.16.36.134:56211)
I20240501 14:00:34.392695 22302 tablet_copy_service.cc:161] P 474d063159a143fcba0416fa01328bc0: Beginning new tablet copy session on tablet d4716a6313a541b981e5d27b01657956 from peer dedba8a189394baca3188f2a5eddbcd4 at {username='slave'} at 127.16.36.134:56211: session id = dedba8a189394baca3188f2a5eddbcd4-d4716a6313a541b981e5d27b01657956
I20240501 14:00:34.393160 22140 catalog_manager.cc:5484] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index -1 to 98, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:34.396040 22302 tablet_copy_source_session.cc:205] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:34.401580 22542 raft_consensus.cc:2932] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:34.406674 24586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4716a6313a541b981e5d27b01657956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.427026 24586 tablet_copy_client.cc:783] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:34.427445 24586 tablet_copy_client.cc:647] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:34.465449 24586 tablet_copy_client.cc:523] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:34.474653 24586 tablet_bootstrap.cc:492] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap starting.
I20240501 14:00:34.898707 24616 ts_tablet_manager.cc:902] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: Initiating tablet copy from peer 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:34.968602 24616 tablet_copy_client.cc:308] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:33963
I20240501 14:00:35.258457 22302 tablet_copy_service.cc:140] P 474d063159a143fcba0416fa01328bc0: Received BeginTabletCopySession request for tablet e6ac874a7f084076bf03ed4167f967e0 from peer 90f980f10dfd441289a8ad090e23e200 ({username='slave'} at 127.16.36.132:43169)
I20240501 14:00:35.258713 22302 tablet_copy_service.cc:161] P 474d063159a143fcba0416fa01328bc0: Beginning new tablet copy session on tablet e6ac874a7f084076bf03ed4167f967e0 from peer 90f980f10dfd441289a8ad090e23e200 at {username='slave'} at 127.16.36.132:43169: session id = 90f980f10dfd441289a8ad090e23e200-e6ac874a7f084076bf03ed4167f967e0
I20240501 14:00:35.261878 22302 tablet_copy_source_session.cc:205] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:35.263327 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac874a7f084076bf03ed4167f967e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.276986 22281 consensus_queue.cc:237] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.156, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:35.294977 24586 tablet_bootstrap.cc:492] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=102 overwritten=0 applied=101 ignored=0} inserts{seen=786 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:35.295660 24586 tablet_bootstrap.cc:492] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap complete.
I20240501 14:00:35.296015 24586 ts_tablet_manager.cc:1367] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: Time spent bootstrapping tablet: real 0.822s user 0.268s sys 0.170s
I20240501 14:00:35.296545 24616 tablet_copy_client.cc:783] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:35.296880 24586 raft_consensus.cc:348] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:35.297924 24586 raft_consensus.cc:729] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedba8a189394baca3188f2a5eddbcd4, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.298301 24586 consensus_queue.cc:260] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 1.102, Last appended by leader: 102, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:35.299858 24616 tablet_copy_client.cc:647] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:35.302795 24616 tablet_copy_client.cc:523] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:35.305963 24586 ts_tablet_manager.cc:1398] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20240501 14:00:35.307533 22302 tablet_copy_service.cc:342] P 474d063159a143fcba0416fa01328bc0: Request end of tablet copy session dedba8a189394baca3188f2a5eddbcd4-d4716a6313a541b981e5d27b01657956 received from {username='slave'} at 127.16.36.134:56211
I20240501 14:00:35.307734 22302 tablet_copy_service.cc:433] P 474d063159a143fcba0416fa01328bc0: ending tablet copy session dedba8a189394baca3188f2a5eddbcd4-d4716a6313a541b981e5d27b01657956 on tablet d4716a6313a541b981e5d27b01657956 with peer dedba8a189394baca3188f2a5eddbcd4
I20240501 14:00:35.332402 24616 tablet_bootstrap.cc:492] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: Bootstrap starting.
I20240501 14:00:35.338099 23858 raft_consensus.cc:1203] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.89->[1.90-1.113] Dedup: 1.102->[1.103-1.113]
W20240501 14:00:35.347134 22218 consensus_peers.cc:485] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 -> Peer bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135:41675): Couldn't send request to peer bcf471e1f6eb42b8a9ae457e6239b89f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bcbcadfd9a34282950350f04b7fcce5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:35.350163 22413 raft_consensus.cc:1261] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240501 14:00:35.356086 24499 consensus_queue.cc:1035] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240501 14:00:35.374332 24431 raft_consensus.cc:2932] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Committing config change with OpId 1.157: config changed from index -1 to 157, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:35.416787 22140 catalog_manager.cc:5484] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index -1 to 157, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:35.429414 22413 raft_consensus.cc:2932] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index -1 to 157, VOTER bcf471e1f6eb42b8a9ae457e6239b89f (127.16.36.135) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:35.720075 22413 consensus_queue.cc:237] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 1.137, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:35.726248 22281 raft_consensus.cc:1261] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
W20240501 14:00:35.729038 22349 consensus_peers.cc:485] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c -> Peer dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555): Couldn't send request to peer dedba8a189394baca3188f2a5eddbcd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe968af0ff994a9093beca7a614e77b7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:35.737315 24361 consensus_queue.cc:1035] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 136, Time since last communication: 0.000s
I20240501 14:00:35.769296 24549 raft_consensus.cc:2932] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.138: config changed from index -1 to 138, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:35.775569 22281 raft_consensus.cc:2932] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index -1 to 138, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:35.782675 22140 catalog_manager.cc:5484] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index -1 to 138, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:35.800400 24667 ts_tablet_manager.cc:902] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: Initiating tablet copy from peer 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:00:35.810669 24667 tablet_copy_client.cc:308] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:33963
I20240501 14:00:35.821043 24616 tablet_bootstrap.cc:492] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=107 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:35.821592 24616 tablet_bootstrap.cc:492] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: Bootstrap complete.
I20240501 14:00:35.821934 24616 ts_tablet_manager.cc:1367] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: Time spent bootstrapping tablet: real 0.490s user 0.168s sys 0.035s
I20240501 14:00:35.822795 24616 raft_consensus.cc:348] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:35.823035 24616 raft_consensus.cc:729] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f980f10dfd441289a8ad090e23e200, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.823349 24616 consensus_queue.cc:260] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:35.845507 24616 ts_tablet_manager.cc:1398] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200: Time spent starting tablet: real 0.023s user 0.007s sys 0.001s
I20240501 14:00:35.846884 22302 tablet_copy_service.cc:342] P 474d063159a143fcba0416fa01328bc0: Request end of tablet copy session 90f980f10dfd441289a8ad090e23e200-e6ac874a7f084076bf03ed4167f967e0 received from {username='slave'} at 127.16.36.132:43169
I20240501 14:00:35.847016 22302 tablet_copy_service.cc:433] P 474d063159a143fcba0416fa01328bc0: ending tablet copy session 90f980f10dfd441289a8ad090e23e200-e6ac874a7f084076bf03ed4167f967e0 on tablet e6ac874a7f084076bf03ed4167f967e0 with peer 90f980f10dfd441289a8ad090e23e200
I20240501 14:00:35.854581 22302 tablet_copy_service.cc:140] P 474d063159a143fcba0416fa01328bc0: Received BeginTabletCopySession request for tablet 7bcbcadfd9a34282950350f04b7fcce5 from peer bcf471e1f6eb42b8a9ae457e6239b89f ({username='slave'} at 127.16.36.135:53933)
I20240501 14:00:35.854811 22302 tablet_copy_service.cc:161] P 474d063159a143fcba0416fa01328bc0: Beginning new tablet copy session on tablet 7bcbcadfd9a34282950350f04b7fcce5 from peer bcf471e1f6eb42b8a9ae457e6239b89f at {username='slave'} at 127.16.36.135:53933: session id = bcf471e1f6eb42b8a9ae457e6239b89f-7bcbcadfd9a34282950350f04b7fcce5
I20240501 14:00:35.857719 22302 tablet_copy_source_session.cc:205] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:35.891125 24667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcbcadfd9a34282950350f04b7fcce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.901001 24667 tablet_copy_client.cc:783] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:35.901372 24667 tablet_copy_client.cc:647] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:35.917764 23591 raft_consensus.cc:1203] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.97->[1.98-1.108] Dedup: 1.107->[1.108-1.108]
I20240501 14:00:35.951597 24667 tablet_copy_client.cc:523] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:35.956980 24667 tablet_bootstrap.cc:492] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap starting.
I20240501 14:00:36.184715 24687 ts_tablet_manager.cc:902] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:36.636016 24687 tablet_copy_client.cc:308] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:00:36.653316 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet fe968af0ff994a9093beca7a614e77b7 from peer dedba8a189394baca3188f2a5eddbcd4 ({username='slave'} at 127.16.36.134:57731)
I20240501 14:00:36.653579 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet fe968af0ff994a9093beca7a614e77b7 from peer dedba8a189394baca3188f2a5eddbcd4 at {username='slave'} at 127.16.36.134:57731: session id = dedba8a189394baca3188f2a5eddbcd4-fe968af0ff994a9093beca7a614e77b7
I20240501 14:00:36.658341 22433 tablet_copy_source_session.cc:205] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:36.664621 24687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe968af0ff994a9093beca7a614e77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.680807 24687 tablet_copy_client.cc:783] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:36.681161 24687 tablet_copy_client.cc:647] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:36.689146 22411 consensus_queue.cc:237] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 145, Committed index: 145, Last appended: 1.146, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:36.708838 24667 tablet_bootstrap.cc:492] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap replayed 1/1 log segments. Stats: ops{read=166 overwritten=0 applied=165 ignored=0} inserts{seen=850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:36.709476 24667 tablet_bootstrap.cc:492] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: Bootstrap complete.
I20240501 14:00:36.712836 24687 tablet_copy_client.cc:523] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:36.713060 24667 ts_tablet_manager.cc:1367] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent bootstrapping tablet: real 0.756s user 0.334s sys 0.126s
I20240501 14:00:36.714387 24667 raft_consensus.cc:348] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:36.715565 24667 raft_consensus.cc:729] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf471e1f6eb42b8a9ae457e6239b89f, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.715898 24667 consensus_queue.cc:260] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 165, Last appended: 1.166, Last appended by leader: 166, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:36.716674 24687 tablet_bootstrap.cc:492] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap starting.
I20240501 14:00:36.747231 24667 ts_tablet_manager.cc:1398] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f: Time spent starting tablet: real 0.034s user 0.006s sys 0.024s
W20240501 14:00:36.748234 22346 consensus_peers.cc:485] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c -> Peer 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725): Couldn't send request to peer 90f980f10dfd441289a8ad090e23e200. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83ae513a62f345d1a6eac152a56936d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:36.749048 22302 tablet_copy_service.cc:342] P 474d063159a143fcba0416fa01328bc0: Request end of tablet copy session bcf471e1f6eb42b8a9ae457e6239b89f-7bcbcadfd9a34282950350f04b7fcce5 received from {username='slave'} at 127.16.36.135:53933
I20240501 14:00:36.749194 22302 tablet_copy_service.cc:433] P 474d063159a143fcba0416fa01328bc0: ending tablet copy session bcf471e1f6eb42b8a9ae457e6239b89f-7bcbcadfd9a34282950350f04b7fcce5 on tablet 7bcbcadfd9a34282950350f04b7fcce5 with peer bcf471e1f6eb42b8a9ae457e6239b89f
W20240501 14:00:36.752550 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:45725, user_credentials={real_user=slave}} blocked reactor thread for 90256.5us
I20240501 14:00:36.765965 22542 raft_consensus.cc:1261] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch)
I20240501 14:00:36.772379 24637 consensus_queue.cc:1035] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 145, Time since last communication: 0.000s
I20240501 14:00:36.806324 24637 raft_consensus.cc:2932] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.147: config changed from index -1 to 147, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:36.816083 22540 raft_consensus.cc:2932] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index -1 to 147, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:36.841518 22140 catalog_manager.cc:5484] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index -1 to 147, VOTER 90f980f10dfd441289a8ad090e23e200 (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:37.181864 24687 tablet_bootstrap.cc:492] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=143 overwritten=0 applied=142 ignored=0} inserts{seen=827 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:00:37.182408 24687 tablet_bootstrap.cc:492] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: Bootstrap complete.
I20240501 14:00:37.182847 24687 ts_tablet_manager.cc:1367] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: Time spent bootstrapping tablet: real 0.466s user 0.208s sys 0.061s
I20240501 14:00:37.183842 24687 raft_consensus.cc:348] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:37.185276 24687 raft_consensus.cc:729] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedba8a189394baca3188f2a5eddbcd4, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.185581 24687 consensus_queue.cc:260] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 142, Last appended: 1.143, Last appended by leader: 143, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:37.214653 24687 ts_tablet_manager.cc:1398] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4: Time spent starting tablet: real 0.032s user 0.005s sys 0.004s
I20240501 14:00:37.216092 24731 ts_tablet_manager.cc:902] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:00:37.222589 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session dedba8a189394baca3188f2a5eddbcd4-fe968af0ff994a9093beca7a614e77b7 received from {username='slave'} at 127.16.36.134:57731
I20240501 14:00:37.222802 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session dedba8a189394baca3188f2a5eddbcd4-fe968af0ff994a9093beca7a614e77b7 on tablet fe968af0ff994a9093beca7a614e77b7 with peer dedba8a189394baca3188f2a5eddbcd4
I20240501 14:00:37.231973 23990 raft_consensus.cc:1203] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.156->[1.157-1.179] Dedup: 1.166->[1.167-1.179]
I20240501 14:00:37.233443 23857 raft_consensus.cc:1203] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.137->[1.138-1.148] Dedup: 1.143->[1.144-1.148]
I20240501 14:00:37.307498 24731 tablet_copy_client.cc:308] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:00:37.318544 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet 83ae513a62f345d1a6eac152a56936d0 from peer 90f980f10dfd441289a8ad090e23e200 ({username='slave'} at 127.16.36.132:35525)
I20240501 14:00:37.318987 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet 83ae513a62f345d1a6eac152a56936d0 from peer 90f980f10dfd441289a8ad090e23e200 at {username='slave'} at 127.16.36.132:35525: session id = 90f980f10dfd441289a8ad090e23e200-83ae513a62f345d1a6eac152a56936d0
I20240501 14:00:37.358290 22433 tablet_copy_source_session.cc:205] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:37.425323 24731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ae513a62f345d1a6eac152a56936d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.432340 24731 tablet_copy_client.cc:783] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:37.432621 24731 tablet_copy_client.cc:647] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:37.474963 24731 tablet_copy_client.cc:523] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:37.480085 24731 tablet_bootstrap.cc:492] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: Bootstrap starting.
I20240501 14:00:37.876470 24731 tablet_bootstrap.cc:492] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: Bootstrap replayed 1/1 log segments. Stats: ops{read=154 overwritten=0 applied=154 ignored=0} inserts{seen=826 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:37.876956 24731 tablet_bootstrap.cc:492] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: Bootstrap complete.
I20240501 14:00:37.877269 24731 ts_tablet_manager.cc:1367] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: Time spent bootstrapping tablet: real 0.397s user 0.247s sys 0.070s
I20240501 14:00:37.878146 24731 raft_consensus.cc:348] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:37.878494 24731 raft_consensus.cc:729] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f980f10dfd441289a8ad090e23e200, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.878835 24731 consensus_queue.cc:260] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 154, Last appended: 1.154, Last appended by leader: 154, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:37.899354 24731 ts_tablet_manager.cc:1398] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200: Time spent starting tablet: real 0.022s user 0.012s sys 0.011s
I20240501 14:00:37.910917 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session 90f980f10dfd441289a8ad090e23e200-83ae513a62f345d1a6eac152a56936d0 received from {username='slave'} at 127.16.36.132:35525
I20240501 14:00:37.911103 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session 90f980f10dfd441289a8ad090e23e200-83ae513a62f345d1a6eac152a56936d0 on tablet 83ae513a62f345d1a6eac152a56936d0 with peer 90f980f10dfd441289a8ad090e23e200
W20240501 14:00:38.146828 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:33963, user_credentials={real_user=slave}} blocked reactor thread for 50836.7us
I20240501 14:00:38.169585 23592 raft_consensus.cc:1203] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.146->[1.147-1.158] Dedup: 1.154->[1.155-1.158]
I20240501 14:00:38.355631 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:38.412036 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:38.468835 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:38.651364 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:38.687656 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:38.749898 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:38.793409 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:38.833884 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:39481, user_credentials={real_user=slave}} blocked reactor thread for 60962.5us
I20240501 14:00:39.444792 22413 consensus_queue.cc:237] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 189, Committed index: 189, Last appended: 1.189, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:39.449731 23856 raft_consensus.cc:1261] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 189. Preceding OpId from leader: term: 1 index: 190. (index mismatch)
I20240501 14:00:39.450642 24808 consensus_queue.cc:1035] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 189, Time since last communication: 0.000s
I20240501 14:00:39.454380 24808 raft_consensus.cc:2932] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.190: config changed from index 138 to 190, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:39.461212 23856 raft_consensus.cc:2932] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Committing config change with OpId 1.190: config changed from index 138 to 190, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:39.468350 22139 catalog_manager.cc:5484] T fe968af0ff994a9093beca7a614e77b7 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 138 to 190, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:39.554075 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet fe968af0ff994a9093beca7a614e77b7 with delete_type TABLET_DATA_TOMBSTONED (TS 474d063159a143fcba0416fa01328bc0 not found in new config with opid_index 190) from {username='slave'} at 127.0.0.1:35434
I20240501 14:00:39.558431 24842 tablet_replica.cc:330] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: stopping tablet replica
I20240501 14:00:39.563741 24842 raft_consensus.cc:2229] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.564517 24842 raft_consensus.cc:2258] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.566586 24842 ts_tablet_manager.cc:1875] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:39.579497 24842 ts_tablet_manager.cc:1888] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.189
I20240501 14:00:39.589785 24842 log.cc:1196] T fe968af0ff994a9093beca7a614e77b7 P 474d063159a143fcba0416fa01328bc0: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe968af0ff994a9093beca7a614e77b7
I20240501 14:00:39.591699 22125 catalog_manager.cc:4830] TS 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): tablet fe968af0ff994a9093beca7a614e77b7 (table rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]) successfully deleted
I20240501 14:00:40.535028 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:00:40.562254 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:33963, user_credentials={real_user=slave}} blocked reactor thread for 63239.1us
I20240501 14:00:40.592025 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:40.638079 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:40.653707 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:40.674613 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:40.692973 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:40.706574 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:00:41.168982 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:38305, user_credentials={real_user=slave}} blocked reactor thread for 43895us
I20240501 14:00:41.714604 22412 consensus_queue.cc:237] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.202, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:41.740465 23592 raft_consensus.cc:1261] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch)
I20240501 14:00:41.743078 24865 consensus_queue.cc:1035] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240501 14:00:41.748358 24917 raft_consensus.cc:2932] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.203: config changed from index 147 to 203, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:41.752436 23592 raft_consensus.cc:2932] T 83ae513a62f345d1a6eac152a56936d0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index 147 to 203, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:41.757812 22139 catalog_manager.cc:5484] T 83ae513a62f345d1a6eac152a56936d0 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 147 to 203, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:42.096570 22523 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ae513a62f345d1a6eac152a56936d0 with delete_type TABLET_DATA_TOMBSTONED (TS 3b1653dc15114e7c873f9cc1a9a3c0a5 not found in new config with opid_index 203) from {username='slave'} at 127.0.0.1:56510
I20240501 14:00:42.121773 24945 tablet_replica.cc:330] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: stopping tablet replica
I20240501 14:00:42.123836 24945 raft_consensus.cc:2229] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:42.127414 24945 raft_consensus.cc:2258] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:00:42.132652 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:33963, user_credentials={real_user=slave}} blocked reactor thread for 52055.6us
I20240501 14:00:42.161362 24945 ts_tablet_manager.cc:1875] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:42.204042 24945 ts_tablet_manager.cc:1888] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.202
I20240501 14:00:42.212107 24945 log.cc:1196] T 83ae513a62f345d1a6eac152a56936d0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/83ae513a62f345d1a6eac152a56936d0
I20240501 14:00:42.229952 22125 catalog_manager.cc:4830] TS 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305): tablet 83ae513a62f345d1a6eac152a56936d0 (table rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]) successfully deleted
I20240501 14:00:42.946859 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:42.963749 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:42.969682 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:42.991549 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:43.002266 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:43.072355 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:43.133538 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:45725, user_credentials={real_user=slave}} blocked reactor thread for 47912.7us
I20240501 14:00:43.232713 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:43.481838 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:39481, user_credentials={real_user=slave}} blocked reactor thread for 87587.5us
I20240501 14:00:43.878561 22280 consensus_queue.cc:237] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:43.892892 23990 raft_consensus.cc:1261] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240501 14:00:43.896996 24923 consensus_queue.cc:1035] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240501 14:00:43.903338 24927 raft_consensus.cc:2932] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 157 to 303, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:43.913627 22139 catalog_manager.cc:5484] T 7bcbcadfd9a34282950350f04b7fcce5 P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index 157 to 303, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:43.919644 23990 raft_consensus.cc:2932] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 157 to 303, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:44.029826 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bcbcadfd9a34282950350f04b7fcce5 with delete_type TABLET_DATA_TOMBSTONED (TS a5574cf7d8b54da09d7f589c2c11e97c not found in new config with opid_index 303) from {username='slave'} at 127.0.0.1:41234
I20240501 14:00:44.045656 25042 tablet_replica.cc:330] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: stopping tablet replica
I20240501 14:00:44.047386 25042 raft_consensus.cc:2229] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.047852 25042 pending_rounds.cc:62] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Trying to abort 1 pending ops.
I20240501 14:00:44.048333 25042 pending_rounds.cc:68] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Aborting op as it isn't in flight: id { term: 1 index: 302 } timestamp: 7022887091616710656 op_type: WRITE_OP write_request { tablet_id: "7bcbcadfd9a34282950350f04b7fcce5" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000S\351\337y\306l\021\302\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""7da081c669501b8b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022887091555921920 authz_token { token_data: "\010\246\230\311\261\006\"3\n\005slave\022*\n e6e5f3c2f86d49159c05269d49a6317a\020\001\030\001 \001(\001" signature: "<redacted>""\027\244\373\230\246\010%`\366\303\302\272\231\213\262w\336~\016\003\262.\302~\230\0266_r\212_)r\373\353\006\200*\362\361\201IM;\2628\000\227_\226\221(Ko\321\353\3605MH\325\214:\326" signing_key_seq_num: 0 } } request_id { client_id: "adf98af3d6ac428c86baa24944809444" seq_no: 863 first_incomplete_seq_no: 863 attempt_no: 0 }
I20240501 14:00:44.049574 25042 raft_consensus.cc:2258] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.056092 25042 ts_tablet_manager.cc:1875] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:44.066521 25042 ts_tablet_manager.cc:1888] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.302
I20240501 14:00:44.066928 25042 log.cc:1196] T 7bcbcadfd9a34282950350f04b7fcce5 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/7bcbcadfd9a34282950350f04b7fcce5
I20240501 14:00:44.068297 22128 catalog_manager.cc:4830] TS a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): tablet 7bcbcadfd9a34282950350f04b7fcce5 (table rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]) successfully deleted
W20240501 14:00:45.028118 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:45725, user_credentials={real_user=slave}} blocked reactor thread for 46997.6us
I20240501 14:00:45.187320 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:45.197646 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:45.206416 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:45.301712 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:45.305184 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:45.329618 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:45.513625 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:00:45.959928 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:45725, user_credentials={real_user=slave}} blocked reactor thread for 49175.3us
I20240501 14:00:46.881621 22281 consensus_queue.cc:237] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 1.276, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:46.890424 23591 raft_consensus.cc:1261] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 276. Preceding OpId from leader: term: 1 index: 277. (index mismatch)
I20240501 14:00:46.895036 25011 consensus_queue.cc:1035] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20240501 14:00:46.959761 24923 raft_consensus.cc:2932] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Committing config change with OpId 1.277: config changed from index 98 to 277, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:46.967365 22139 catalog_manager.cc:5484] T e6ac874a7f084076bf03ed4167f967e0 P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index 98 to 277, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:46.989343 23591 raft_consensus.cc:2932] T e6ac874a7f084076bf03ed4167f967e0 P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Committing config change with OpId 1.277: config changed from index 98 to 277, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:47.470772 22523 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ac874a7f084076bf03ed4167f967e0 with delete_type TABLET_DATA_TOMBSTONED (TS 3b1653dc15114e7c873f9cc1a9a3c0a5 not found in new config with opid_index 277) from {username='slave'} at 127.0.0.1:56510
I20240501 14:00:47.477521 25160 tablet_replica.cc:330] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: stopping tablet replica
I20240501 14:00:47.477859 25160 raft_consensus.cc:2229] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.490947 25160 raft_consensus.cc:2258] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.492074 25160 ts_tablet_manager.cc:1875] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.502453 25160 ts_tablet_manager.cc:1888] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.276
I20240501 14:00:47.502730 25160 log.cc:1196] T e6ac874a7f084076bf03ed4167f967e0 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6ac874a7f084076bf03ed4167f967e0
I20240501 14:00:47.504455 22125 catalog_manager.cc:4830] TS 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305): tablet e6ac874a7f084076bf03ed4167f967e0 (table rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]) successfully deleted
I20240501 14:00:48.356117 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:48.359540 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:48.361578 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:48.363121 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:48.482506 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:48.540735 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:48.619187 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:50.091362 23780 kernel_stack_watchdog.cc:198] Thread 23856 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 251ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f09445fc980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f094126ad11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f094128b7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f09412873ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f094128c4ff kudu::ThreadPool::Submit() at ??:0
@ 0x7f094ddf69c9 kudu::tablet::OpDriver::ApplyAsync() at ??:0
@ 0x7f094ddf780d kudu::tablet::OpDriver::ReplicationFinished() at ??:0
@ 0x7f094dda8544 kudu::tablet::TabletReplica::StartFollowerOp()::$_7::operator()() at ??:0
@ 0x7f094dda83ad std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f095479817e std::function<>::operator()() at ??:0
@ 0x7f094cac305d kudu::consensus::ConsensusRound::NotifyReplicationFinished() at ??:0
@ 0x7f094ca5051d kudu::consensus::PendingRounds::AdvanceCommittedIndex() at ??:0
@ 0x7f094ca99737 kudu::consensus::RaftConsensus::UpdateReplica() at ??:0
I20240501 14:00:50.289620 22282 consensus_queue.cc:237] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 1.262, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:00:50.295321 23857 raft_consensus.cc:1261] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
I20240501 14:00:50.301772 25174 consensus_queue.cc:1035] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20240501 14:00:50.305119 25174 raft_consensus.cc:2932] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 [term 1 LEADER]: Committing config change with OpId 1.263: config changed from index 90 to 263, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:50.310331 23856 raft_consensus.cc:2932] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index 90 to 263, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:50.325282 22139 catalog_manager.cc:5484] T d4716a6313a541b981e5d27b01657956 P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index 90 to 263, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:00:50.615962 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet d4716a6313a541b981e5d27b01657956 with delete_type TABLET_DATA_TOMBSTONED (TS a5574cf7d8b54da09d7f589c2c11e97c not found in new config with opid_index 263) from {username='slave'} at 127.0.0.1:41234
I20240501 14:00:50.625012 25272 tablet_replica.cc:330] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: stopping tablet replica
I20240501 14:00:50.625252 25272 raft_consensus.cc:2229] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.625598 25272 raft_consensus.cc:2258] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.629652 25272 ts_tablet_manager.cc:1875] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:50.651085 25272 ts_tablet_manager.cc:1888] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.262
I20240501 14:00:50.651254 25272 log.cc:1196] T d4716a6313a541b981e5d27b01657956 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/d4716a6313a541b981e5d27b01657956
I20240501 14:00:50.652215 22128 catalog_manager.cc:4830] TS a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): tablet d4716a6313a541b981e5d27b01657956 (table rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]) successfully deleted
I20240501 14:00:52.152164 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:52.171053 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:52.358224 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:52.380164 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:52.381686 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:52.413502 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:52.443291 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:52.920015 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:38305, user_credentials={real_user=slave}} blocked reactor thread for 46983.9us
I20240501 14:00:53.440142 22541 consensus_queue.cc:237] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:00:53.450102 23987 raft_consensus.cc:1261] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20240501 14:00:53.451170 25347 consensus_queue.cc:1035] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 314, Time since last communication: 0.000s
I20240501 14:00:53.461665 24778 raft_consensus.cc:2932] T dd5f6e034ac14cd2864d894fe5b00fd9 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index 100 to 316, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:53.463048 23987 raft_consensus.cc:2932] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 100 to 316, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:53.472716 22137 catalog_manager.cc:5484] T dd5f6e034ac14cd2864d894fe5b00fd9 P bcf471e1f6eb42b8a9ae457e6239b89f reported cstate change: config changed from index 100 to 316, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:00:53.554265 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5f6e034ac14cd2864d894fe5b00fd9 with delete_type TABLET_DATA_TOMBSTONED (TS 474d063159a143fcba0416fa01328bc0 not found in new config with opid_index 316) from {username='slave'} at 127.0.0.1:35434
I20240501 14:00:53.586603 25388 tablet_replica.cc:330] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: stopping tablet replica
I20240501 14:00:53.586973 25388 raft_consensus.cc:2229] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.587358 25388 pending_rounds.cc:62] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Trying to abort 1 pending ops.
I20240501 14:00:53.587589 25388 pending_rounds.cc:68] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Aborting op as it isn't in flight: id { term: 1 index: 314 } timestamp: 7022887130715521024 op_type: WRITE_OP write_request { tablet_id: "dd5f6e034ac14cd2864d894fe5b00fd9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000X\376[~\237\\i\372\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""449821e43fb2042e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022887130662014976 authz_token { token_data: "\010\246\230\311\261\006\"3\n\005slave\022*\n 5373149d652d4066a2b9b99544e79edd\020\001\030\001 \001(\001" signature: "<redacted>""h\234ns\371\346\321T\203\006\2155\222Cj\376g\203\271}\237\217W2\260D\343\326\274\210-\275\317*\375\305I\243;*\366f\330\021t\246\271\344W\307\355k\222n\211\374\330E@\033\216v\235n" signing_key_seq_num: 0 } } request_id { client_id: "e28043df66e44b5282ec7e6b2027ef6a" seq_no: 942 first_incomplete_seq_no: 942 attempt_no: 0 }
I20240501 14:00:53.588743 25388 raft_consensus.cc:2258] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.593446 25388 ts_tablet_manager.cc:1875] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:53.601001 25388 ts_tablet_manager.cc:1888] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.314
I20240501 14:00:53.601297 25388 log.cc:1196] T dd5f6e034ac14cd2864d894fe5b00fd9 P 474d063159a143fcba0416fa01328bc0: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd5f6e034ac14cd2864d894fe5b00fd9
I20240501 14:00:53.602463 22125 catalog_manager.cc:4830] TS 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): tablet dd5f6e034ac14cd2864d894fe5b00fd9 (table rebalance_test_table_1 [id=5373149d652d4066a2b9b99544e79edd]) successfully deleted
I20240501 14:00:54.389345 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:54.477113 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:00:54.513136 23660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 48004.3us
I20240501 14:00:54.592021 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:54.659528 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:54.676852 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:54.766453 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:54.812243 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:54.994249 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:45725, user_credentials={real_user=slave}} blocked reactor thread for 76181.4us
I20240501 14:00:55.083129 22411 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e"
dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45858
I20240501 14:00:55.083506 22411 raft_consensus.cc:595] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Received request to transfer leadership
W20240501 14:00:55.124888 22370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.135040 22370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.155473 22370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.186136 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.202754 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.216874 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.228068 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.241245 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.257892 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.270916 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:39481, user_credentials={real_user=slave}} blocked reactor thread for 69255.5us
W20240501 14:00:55.272831 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.299495 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.315089 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.348676 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
W20240501 14:00:55.370038 22370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
I20240501 14:00:55.388978 25350 raft_consensus.cc:979] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: : Instructing follower 474d063159a143fcba0416fa01328bc0 to start an election
I20240501 14:00:55.390297 25350 raft_consensus.cc:1067] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Signalling peer 474d063159a143fcba0416fa01328bc0 to start an election
W20240501 14:00:55.392678 22370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Service unavailable: leader transfer in progress
I20240501 14:00:55.393234 22281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e"
dest_uuid: "474d063159a143fcba0416fa01328bc0"
from {username='slave'} at 127.16.36.130:53695
I20240501 14:00:55.393766 22281 raft_consensus.cc:482] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:55.393925 22281 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:55.396647 22281 raft_consensus.cc:504] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:55.398010 22281 leader_election.cc:288] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 2 election: Requested vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541), 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481)
I20240501 14:00:55.412375 22411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: true dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
I20240501 14:00:55.412966 22411 raft_consensus.cc:3032] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:55.413239 22411 raft_consensus.cc:729] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:00:55.415330 22411 consensus_queue.cc:260] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 306, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:55.430074 22411 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:55.423882 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Illegal state: replica a5574cf7d8b54da09d7f589c2c11e97c is not leader of this config: current role FOLLOWER
W20240501 14:00:55.456176 22240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42762: Illegal state: replica 474d063159a143fcba0416fa01328bc0 is not leader of this config: current role FOLLOWER
W20240501 14:00:55.488018 22370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Illegal state: replica a5574cf7d8b54da09d7f589c2c11e97c is not leader of this config: current role FOLLOWER
W20240501 14:00:55.510402 22240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42762: Illegal state: replica 474d063159a143fcba0416fa01328bc0 is not leader of this config: current role FOLLOWER
I20240501 14:00:55.572492 23725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e" candidate_uuid: "474d063159a143fcba0416fa01328bc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: true dest_uuid: "5e1c3e3144384f90b47255623d63575b"
I20240501 14:00:55.572930 23725 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:55.577565 23725 raft_consensus.cc:2454] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 2.
I20240501 14:00:55.594554 22215 leader_election.cc:302] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, 5e1c3e3144384f90b47255623d63575b; no voters:
I20240501 14:00:55.604310 25361 raft_consensus.cc:2781] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:55.604566 25361 raft_consensus.cc:686] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 LEADER]: Becoming Leader. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Running, Role: LEADER
I20240501 14:00:55.604982 25361 consensus_queue.cc:237] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 306, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:00:55.613301 22137 catalog_manager.cc:5484] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 reported cstate change: term changed from 1 to 2, leader changed from a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) to 474d063159a143fcba0416fa01328bc0 (127.16.36.129). New cstate: current_term: 2 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
W20240501 14:00:55.641520 23685 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50396: Illegal state: replica 5e1c3e3144384f90b47255623d63575b is not leader of this config: current role FOLLOWER
I20240501 14:00:55.653050 22411 raft_consensus.cc:2454] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474d063159a143fcba0416fa01328bc0 in term 2.
W20240501 14:00:55.681934 22370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38202: Illegal state: replica a5574cf7d8b54da09d7f589c2c11e97c is not leader of this config: current role FOLLOWER
I20240501 14:00:55.743640 23725 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20240501 14:00:55.747941 25482 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20240501 14:00:55.758467 25451 mvcc.cc:205] Tried to move back new op lower bound from 7022887140189089792 to 7022887139776376832. Current Snapshot: MvccSnapshot[applied={T|T < 7022887140189089792}]
I20240501 14:00:55.777050 22413 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20240501 14:00:55.783175 25481 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.002s
I20240501 14:00:56.173466 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:56.214282 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:56.341647 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:56.411214 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:56.440011 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:56.477177 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:56.509692 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:57.169178 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:33963, user_credentials={real_user=slave}} blocked reactor thread for 59152.3us
W20240501 14:00:57.866605 22479 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:35555, user_credentials={real_user=slave}} blocked reactor thread for 54509.7us
I20240501 14:00:58.381624 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:58.551798 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:58.562888 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:58.583565 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:58.589207 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:00:58.605386 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:38305, user_credentials={real_user=slave}} blocked reactor thread for 64143.5us
I20240501 14:00:58.668248 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:58.767485 22215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:38305, user_credentials={real_user=slave}} blocked reactor thread for 104357us
I20240501 14:00:58.780069 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:00:59.220248 23660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 58351.6us
W20240501 14:00:59.270210 22479 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 55991.6us
I20240501 14:00:59.421151 22413 consensus_queue.cc:237] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 1.370, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:00:59.425601 23591 raft_consensus.cc:1261] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 370. Preceding OpId from leader: term: 1 index: 371. (index mismatch)
I20240501 14:00:59.427654 25355 consensus_queue.cc:1035] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 369, Time since last communication: 0.000s
I20240501 14:00:59.431780 25355 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.371: config changed from index 61 to 371, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:59.432698 23591 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Committing config change with OpId 1.371: config changed from index 61 to 371, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:59.439236 22140 catalog_manager.cc:5484] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 61 to 371, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:00:59.497097 22523 tablet_service.cc:1514] Processing DeleteTablet for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf with delete_type TABLET_DATA_TOMBSTONED (TS 3b1653dc15114e7c873f9cc1a9a3c0a5 not found in new config with opid_index 371) from {username='slave'} at 127.0.0.1:56510
I20240501 14:00:59.500170 25636 tablet_replica.cc:330] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: stopping tablet replica
I20240501 14:00:59.500492 25636 raft_consensus.cc:2229] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:59.500838 25636 pending_rounds.cc:62] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Trying to abort 1 pending ops.
I20240501 14:00:59.501005 25636 pending_rounds.cc:68] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Aborting op as it isn't in flight: id { term: 1 index: 370 } timestamp: 7022887155367804928 op_type: WRITE_OP write_request { tablet_id: "c09f45be8c064ecdbdc78d5ce8dbe8bf" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000x\324\006[\334h\332e\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a9d62a22d6d0a8ed" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022887155324100608 authz_token { token_data: "\010\246\230\311\261\006\"3\n\005slave\022*\n aa4393d7657c42b3b15522ad50965d6c\020\001\030\001 \001(\001" signature: "<redacted>""j@\376W\232\315n\302\201\3437/zS\3313\034\t7F\237\240\253\305Jr7\274\246G\305\355hS=wwj\372}Y\002\336\215Q}\327J\025:&\327\372\342\275\2530\212\177}\2156(\206" signing_key_seq_num: 0 } } request_id { client_id: "aee263622d8b46de9226072641598f11" seq_no: 1075 first_incomplete_seq_no: 1075 attempt_no: 0 }
I20240501 14:00:59.518328 25636 raft_consensus.cc:2258] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.521027 25636 ts_tablet_manager.cc:1875] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:59.561344 25636 ts_tablet_manager.cc:1888] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.370
I20240501 14:00:59.561599 25636 log.cc:1196] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:00:59.562862 22125 catalog_manager.cc:4830] TS 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305): tablet c09f45be8c064ecdbdc78d5ce8dbe8bf (table rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]) successfully deleted
I20240501 14:01:00.523931 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:00.528649 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:00.536885 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:00.540563 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:00.569880 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:00.639873 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:00.905000 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:01.011243 23660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 85751.4us
I20240501 14:01:01.748618 22542 consensus_queue.cc:237] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 551, Committed index: 551, Last appended: 1.551, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:01:01.751281 23857 raft_consensus.cc:1261] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 551. Preceding OpId from leader: term: 1 index: 552. (index mismatch)
I20240501 14:01:01.751976 25661 consensus_queue.cc:1035] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 552, Last known committed idx: 551, Time since last communication: 0.000s
I20240501 14:01:01.755385 25641 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.552: config changed from index 60 to 552, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:01.773635 23856 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Committing config change with OpId 1.552: config changed from index 60 to 552, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:01.784586 22139 catalog_manager.cc:5484] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index 60 to 552, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:01.999644 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet c40fd7b8bfd04d8ca67a9acf40f74445 with delete_type TABLET_DATA_TOMBSTONED (TS 474d063159a143fcba0416fa01328bc0 not found in new config with opid_index 552) from {username='slave'} at 127.0.0.1:35434
I20240501 14:01:02.112671 25750 tablet_replica.cc:330] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: stopping tablet replica
I20240501 14:01:02.112911 25750 raft_consensus.cc:2229] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.113118 25750 raft_consensus.cc:2258] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.116392 25750 ts_tablet_manager.cc:1875] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:02.165432 25750 ts_tablet_manager.cc:1888] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.551
I20240501 14:01:02.171492 25750 log.cc:1196] T c40fd7b8bfd04d8ca67a9acf40f74445 P 474d063159a143fcba0416fa01328bc0: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:01:02.173919 22125 catalog_manager.cc:4830] TS 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): tablet c40fd7b8bfd04d8ca67a9acf40f74445 (table rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]) successfully deleted
W20240501 14:01:02.911103 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:33963, user_credentials={real_user=slave}} blocked reactor thread for 48837.4us
I20240501 14:01:03.009256 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:03.027065 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:03.182477 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:03.204777 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:03.271510 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:03.291613 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:03.296586 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:03.325695 22479 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:46541, user_credentials={real_user=slave}} blocked reactor thread for 57587.3us
W20240501 14:01:03.962828 22479 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:35555, user_credentials={real_user=slave}} blocked reactor thread for 44536.9us
I20240501 14:01:04.133291 22542 consensus_queue.cc:237] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 545, Committed index: 545, Last appended: 1.545, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:04.138733 23724 raft_consensus.cc:1261] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 545. Preceding OpId from leader: term: 1 index: 546. (index mismatch)
I20240501 14:01:04.142951 25710 consensus_queue.cc:1035] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 546, Last known committed idx: 545, Time since last communication: 0.000s
I20240501 14:01:04.146862 25710 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.546: config changed from index 79 to 546, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:04.151877 23724 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Committing config change with OpId 1.546: config changed from index 79 to 546, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:04.157711 22139 catalog_manager.cc:5484] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index 79 to 546, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:04.233692 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet b28e7ed3a23243ee98327f60b2ebdbd4 with delete_type TABLET_DATA_TOMBSTONED (TS a5574cf7d8b54da09d7f589c2c11e97c not found in new config with opid_index 546) from {username='slave'} at 127.0.0.1:41234
I20240501 14:01:04.324388 25840 tablet_replica.cc:330] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: stopping tablet replica
I20240501 14:01:04.324733 25840 raft_consensus.cc:2229] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.325105 25840 raft_consensus.cc:2258] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.326562 25840 ts_tablet_manager.cc:1875] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.346346 25840 ts_tablet_manager.cc:1888] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.545
I20240501 14:01:04.346602 25840 log.cc:1196] T b28e7ed3a23243ee98327f60b2ebdbd4 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:01:04.347724 22128 catalog_manager.cc:4830] TS a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): tablet b28e7ed3a23243ee98327f60b2ebdbd4 (table rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]) successfully deleted
I20240501 14:01:05.357208 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:05.397643 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:05.459055 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:05.491922 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:05.515857 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:05.555855 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:05.607873 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:06.468500 22412 consensus_queue.cc:237] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 451, Committed index: 451, Last appended: 1.451, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 452 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:01:06.473006 23988 raft_consensus.cc:1261] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 451. Preceding OpId from leader: term: 1 index: 452. (index mismatch)
I20240501 14:01:06.474948 25802 consensus_queue.cc:1035] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 452, Last known committed idx: 451, Time since last communication: 0.000s
I20240501 14:01:06.480990 25866 raft_consensus.cc:2932] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.452: config changed from index 64 to 452, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 452 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:01:06.486697 23988 raft_consensus.cc:2932] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Committing config change with OpId 1.452: config changed from index 64 to 452, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 452 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:01:06.497207 22139 catalog_manager.cc:5484] T 161faaa672ce4a0e80495369f1060558 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 64 to 452, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 452 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } } }
I20240501 14:01:06.572286 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet 161faaa672ce4a0e80495369f1060558 with delete_type TABLET_DATA_TOMBSTONED (TS 474d063159a143fcba0416fa01328bc0 not found in new config with opid_index 452) from {username='slave'} at 127.0.0.1:35434
I20240501 14:01:06.587181 25945 tablet_replica.cc:330] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: stopping tablet replica
I20240501 14:01:06.587503 25945 raft_consensus.cc:2229] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:06.587972 25945 raft_consensus.cc:2258] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:06.601343 25945 ts_tablet_manager.cc:1875] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:06.607671 25945 ts_tablet_manager.cc:1888] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.451
I20240501 14:01:06.607789 25945 log.cc:1196] T 161faaa672ce4a0e80495369f1060558 P 474d063159a143fcba0416fa01328bc0: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/161faaa672ce4a0e80495369f1060558
I20240501 14:01:06.616338 22125 catalog_manager.cc:4830] TS 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): tablet 161faaa672ce4a0e80495369f1060558 (table rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]) successfully deleted
I20240501 14:01:07.865582 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:07.904574 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:07.911203 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:07.932628 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:07.951954 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:07.990339 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:08.099308 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:01:08.358191 22479 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 54939us
I20240501 14:01:08.714457 22412 consensus_queue.cc:237] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 502, Committed index: 502, Last appended: 1.502, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:01:08.717047 23858 raft_consensus.cc:1261] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240501 14:01:08.718962 25355 consensus_queue.cc:1035] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240501 14:01:08.722004 25802 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.503: config changed from index 88 to 503, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:08.727788 23856 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 88 to 503, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:08.728973 22137 catalog_manager.cc:5484] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 88 to 503, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:08.778440 22523 tablet_service.cc:1514] Processing DeleteTablet for tablet 999759caca8344f2b2ae0cff5e204425 with delete_type TABLET_DATA_TOMBSTONED (TS 3b1653dc15114e7c873f9cc1a9a3c0a5 not found in new config with opid_index 503) from {username='slave'} at 127.0.0.1:56510
W20240501 14:01:08.796068 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:45725, user_credentials={real_user=slave}} blocked reactor thread for 60599.5us
I20240501 14:01:08.799413 26035 tablet_replica.cc:330] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: stopping tablet replica
I20240501 14:01:08.799629 26035 raft_consensus.cc:2229] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:08.799827 26035 raft_consensus.cc:2258] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:08.801481 26035 ts_tablet_manager.cc:1875] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:08.809553 26035 ts_tablet_manager.cc:1888] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.502
I20240501 14:01:08.809703 26035 log.cc:1196] T 999759caca8344f2b2ae0cff5e204425 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/999759caca8344f2b2ae0cff5e204425
I20240501 14:01:08.811707 22125 catalog_manager.cc:4830] TS 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305): tablet 999759caca8344f2b2ae0cff5e204425 (table rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]) successfully deleted
I20240501 14:01:09.885327 22523 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:09.966768 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:10.016536 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:10.017441 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:10.025542 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:10.048355 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:10.157454 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:10.616393 22542 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74e03b7d269c40ff9fbf6457611d849b"
dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47518
I20240501 14:01:10.616693 22542 raft_consensus.cc:595] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:01:10.621048 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.626925 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.636128 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.645510 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.656623 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.665674 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.680469 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.693331 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.711444 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.728209 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.742200 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.755740 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.772490 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.791379 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.810703 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.829499 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.848299 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.868590 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.899631 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.923574 22501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
W20240501 14:01:10.960719 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
I20240501 14:01:10.983983 25916 raft_consensus.cc:979] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: : Instructing follower a5574cf7d8b54da09d7f589c2c11e97c to start an election
I20240501 14:01:10.984194 25916 raft_consensus.cc:1067] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Signalling peer a5574cf7d8b54da09d7f589c2c11e97c to start an election
W20240501 14:01:10.985813 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Service unavailable: leader transfer in progress
I20240501 14:01:10.985937 22412 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74e03b7d269c40ff9fbf6457611d849b"
dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
from {username='slave'} at 127.16.36.131:40087
I20240501 14:01:10.986210 22412 raft_consensus.cc:482] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:10.986331 22412 raft_consensus.cc:3037] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:11.000031 22412 raft_consensus.cc:504] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:01:11.004010 22412 leader_election.cc:288] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 2 election: Requested vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305), 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725)
I20240501 14:01:11.005470 22542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e03b7d269c40ff9fbf6457611d849b" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 2 candidate_status { last_received { term: 1 index: 765 } } ignore_live_leader: true dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
I20240501 14:01:11.005736 22542 raft_consensus.cc:3032] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:11.005852 22542 raft_consensus.cc:729] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Running, Role: LEADER
I20240501 14:01:11.006188 22542 consensus_queue.cc:260] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 765, Committed index: 765, Last appended: 1.765, Last appended by leader: 765, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:01:11.006774 22542 raft_consensus.cc:3037] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:11.006783 23591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e03b7d269c40ff9fbf6457611d849b" candidate_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" candidate_term: 2 candidate_status { last_received { term: 1 index: 765 } } ignore_live_leader: true dest_uuid: "90f980f10dfd441289a8ad090e23e200"
I20240501 14:01:11.007138 23591 raft_consensus.cc:3037] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:11.009631 22542 raft_consensus.cc:2454] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 2.
I20240501 14:01:11.010663 23591 raft_consensus.cc:2454] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5574cf7d8b54da09d7f589c2c11e97c in term 2.
W20240501 14:01:11.013216 22502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37418: Illegal state: replica 3b1653dc15114e7c873f9cc1a9a3c0a5 is not leader of this config: current role FOLLOWER
I20240501 14:01:11.015349 22346 leader_election.cc:302] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f980f10dfd441289a8ad090e23e200, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:01:11.015934 25355 raft_consensus.cc:2781] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:11.016110 25355 raft_consensus.cc:686] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 2 LEADER]: Becoming Leader. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:01:11.016424 25355 consensus_queue.cc:237] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 765, Committed index: 765, Last appended: 1.765, Last appended by leader: 765, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:01:11.020190 22137 catalog_manager.cc:5484] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: term changed from 1 to 2, leader changed from 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) to a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130). New cstate: current_term: 2 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:01:11.192402 23589 raft_consensus.cc:1261] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 765. Preceding OpId from leader: term: 2 index: 767. (index mismatch)
I20240501 14:01:11.193536 22542 raft_consensus.cc:1261] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 765. Preceding OpId from leader: term: 2 index: 767. (index mismatch)
I20240501 14:01:11.198470 25545 consensus_queue.cc:1035] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 766, Last known committed idx: 765, Time since last communication: 0.000s
I20240501 14:01:11.198989 25866 consensus_queue.cc:1035] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 766, Last known committed idx: 765, Time since last communication: 0.000s
I20240501 14:01:11.556798 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:11.601724 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:11.635802 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:11.639367 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:11.654150 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:11.669217 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:11.837746 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:11.952702 23660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 45580.9us
I20240501 14:01:12.532002 22411 consensus_queue.cc:237] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 785, Committed index: 785, Last appended: 2.785, Last appended by leader: 765, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:01:12.539191 23590 raft_consensus.cc:1261] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 2 index: 785. Preceding OpId from leader: term: 2 index: 786. (index mismatch)
I20240501 14:01:12.540940 25866 consensus_queue.cc:1035] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 786, Last known committed idx: 785, Time since last communication: 0.000s
I20240501 14:01:12.554011 25866 raft_consensus.cc:2932] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c [term 2 LEADER]: Committing config change with OpId 2.786: config changed from index 98 to 786, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:01:12.555349 23589 raft_consensus.cc:2932] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Committing config change with OpId 2.786: config changed from index 98 to 786, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:01:12.575323 22137 catalog_manager.cc:5484] T 74e03b7d269c40ff9fbf6457611d849b P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 98 to 786, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } }
I20240501 14:01:12.621186 22523 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e03b7d269c40ff9fbf6457611d849b with delete_type TABLET_DATA_TOMBSTONED (TS 3b1653dc15114e7c873f9cc1a9a3c0a5 not found in new config with opid_index 786) from {username='slave'} at 127.0.0.1:56510
I20240501 14:01:12.623123 26200 tablet_replica.cc:330] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: stopping tablet replica
I20240501 14:01:12.623443 26200 raft_consensus.cc:2229] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.624121 26200 raft_consensus.cc:2258] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.625311 26200 ts_tablet_manager.cc:1875] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.643505 26200 ts_tablet_manager.cc:1888] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.785
I20240501 14:01:12.643786 26200 log.cc:1196] T 74e03b7d269c40ff9fbf6457611d849b P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/74e03b7d269c40ff9fbf6457611d849b
I20240501 14:01:12.644908 22125 catalog_manager.cc:4830] TS 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305): tablet 74e03b7d269c40ff9fbf6457611d849b (table rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]) successfully deleted
I20240501 14:01:14.599627 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:14.608997 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:14.612051 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:14.650131 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:14.656488 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:14.672618 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:14.794198 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:15.309687 22282 consensus_queue.cc:237] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 576, Committed index: 576, Last appended: 2.576, Last appended by leader: 306, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:15.313248 23726 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 2 index: 576. Preceding OpId from leader: term: 2 index: 577. (index mismatch)
I20240501 14:01:15.314659 26096 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 577, Last known committed idx: 576, Time since last communication: 0.000s
I20240501 14:01:15.317471 26096 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 LEADER]: Committing config change with OpId 2.577: config changed from index 77 to 577, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:15.324460 23726 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Committing config change with OpId 2.577: config changed from index 77 to 577, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:15.333341 22139 catalog_manager.cc:5484] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index 77 to 577, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New cstate: current_term: 2 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:15.407721 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet dc19025daec44883afec64b3d12c7c4e with delete_type TABLET_DATA_TOMBSTONED (TS a5574cf7d8b54da09d7f589c2c11e97c not found in new config with opid_index 577) from {username='slave'} at 127.0.0.1:41234
I20240501 14:01:15.417851 26338 tablet_replica.cc:330] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: stopping tablet replica
I20240501 14:01:15.418394 26338 raft_consensus.cc:2229] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:15.418716 26338 raft_consensus.cc:2258] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:15.419924 26338 ts_tablet_manager.cc:1875] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:15.426211 26338 ts_tablet_manager.cc:1888] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.576
I20240501 14:01:15.426445 26338 log.cc:1196] T dc19025daec44883afec64b3d12c7c4e P a5574cf7d8b54da09d7f589c2c11e97c: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc19025daec44883afec64b3d12c7c4e
I20240501 14:01:15.427500 22128 catalog_manager.cc:4830] TS a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): tablet dc19025daec44883afec64b3d12c7c4e (table rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]) successfully deleted
I20240501 14:01:16.887439 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:16.893263 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:16.995872 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:17.057440 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:17.067354 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:17.094668 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:17.160882 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:17.558188 22282 consensus_queue.cc:237] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 614, Committed index: 614, Last appended: 2.615, Last appended by leader: 306, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
W20240501 14:01:17.562119 22218 consensus_peers.cc:485] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 -> Peer dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555): Couldn't send request to peer dedba8a189394baca3188f2a5eddbcd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc19025daec44883afec64b3d12c7c4e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.563699 23726 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Refusing update from remote peer 474d063159a143fcba0416fa01328bc0: Log matching property violated. Preceding OpId in replica: term: 2 index: 615. Preceding OpId from leader: term: 2 index: 616. (index mismatch)
I20240501 14:01:17.572180 26217 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 616, Last known committed idx: 614, Time since last communication: 0.000s
I20240501 14:01:17.577482 26217 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 LEADER]: Committing config change with OpId 2.616: config changed from index 577 to 616, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:17.580725 23726 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Committing config change with OpId 2.616: config changed from index 577 to 616, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New config: { opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:17.585669 22137 catalog_manager.cc:5484] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 reported cstate change: config changed from index 577 to 616, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) added. New cstate: current_term: 2 leader_uuid: "474d063159a143fcba0416fa01328bc0" committed_config { opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:18.030470 26491 ts_tablet_manager.cc:902] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: Initiating tablet copy from peer 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
I20240501 14:01:18.037339 22411 consensus_queue.cc:237] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 658, Committed index: 658, Last appended: 1.658, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:18.039920 23857 raft_consensus.cc:1261] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 658. Preceding OpId from leader: term: 1 index: 659. (index mismatch)
I20240501 14:01:18.043044 26481 consensus_queue.cc:1035] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 659, Last known committed idx: 658, Time since last communication: 0.000s
I20240501 14:01:18.045717 26481 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.659: config changed from index 503 to 659, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New config: { opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:18.055456 22139 catalog_manager.cc:5484] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 503 to 659, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:18.069438 23854 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Committing config change with OpId 1.659: config changed from index 503 to 659, VOTER 5e1c3e3144384f90b47255623d63575b (127.16.36.133) added. New config: { opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
W20240501 14:01:18.074546 22346 consensus_peers.cc:485] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c -> Peer 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481): Couldn't send request to peer 5e1c3e3144384f90b47255623d63575b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 999759caca8344f2b2ae0cff5e204425. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.121356 26491 tablet_copy_client.cc:308] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:33963
I20240501 14:01:18.124218 22302 tablet_copy_service.cc:140] P 474d063159a143fcba0416fa01328bc0: Received BeginTabletCopySession request for tablet dc19025daec44883afec64b3d12c7c4e from peer dedba8a189394baca3188f2a5eddbcd4 ({username='slave'} at 127.16.36.134:56211)
I20240501 14:01:18.124591 22302 tablet_copy_service.cc:161] P 474d063159a143fcba0416fa01328bc0: Beginning new tablet copy session on tablet dc19025daec44883afec64b3d12c7c4e from peer dedba8a189394baca3188f2a5eddbcd4 at {username='slave'} at 127.16.36.134:56211: session id = dedba8a189394baca3188f2a5eddbcd4-dc19025daec44883afec64b3d12c7c4e
I20240501 14:01:18.128378 22302 tablet_copy_source_session.cc:205] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.149277 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19025daec44883afec64b3d12c7c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.158130 26491 tablet_copy_client.cc:783] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.160144 26491 tablet_copy_client.cc:647] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.197329 26491 tablet_copy_client.cc:523] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.213516 26491 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: Bootstrap starting.
I20240501 14:01:18.491202 26510 ts_tablet_manager.cc:902] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:01:18.692513 26510 tablet_copy_client.cc:308] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:01:18.693459 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet 999759caca8344f2b2ae0cff5e204425 from peer 5e1c3e3144384f90b47255623d63575b ({username='slave'} at 127.16.36.133:45659)
I20240501 14:01:18.693617 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet 999759caca8344f2b2ae0cff5e204425 from peer 5e1c3e3144384f90b47255623d63575b at {username='slave'} at 127.16.36.133:45659: session id = 5e1c3e3144384f90b47255623d63575b-999759caca8344f2b2ae0cff5e204425
I20240501 14:01:18.697734 22433 tablet_copy_source_session.cc:205] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:18.700700 26510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999759caca8344f2b2ae0cff5e204425. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.715058 26510 tablet_copy_client.cc:783] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.718740 26510 tablet_copy_client.cc:647] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.725477 26510 tablet_copy_client.cc:523] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.730031 26510 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: Bootstrap starting.
I20240501 14:01:19.406529 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:19.453512 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:19.589859 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:19.778162 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:19.781963 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:19.871361 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:01:19.953374 26491 log.cc:924] Time spent T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20240501 14:01:20.237493 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:20.449182 22218 consensus_peers.cc:485] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 -> Peer dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555): Couldn't send request to peer dedba8a189394baca3188f2a5eddbcd4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:20.914613 22346 consensus_peers.cc:485] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c -> Peer 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481): Couldn't send request to peer 5e1c3e3144384f90b47255623d63575b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:21.115468 26510 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: Bootstrap replayed 1/1 log segments. Stats: ops{read=667 overwritten=0 applied=667 ignored=0} inserts{seen=1337 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:21.116142 26510 tablet_bootstrap.cc:492] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: Bootstrap complete.
I20240501 14:01:21.116606 26510 ts_tablet_manager.cc:1367] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: Time spent bootstrapping tablet: real 2.387s user 0.964s sys 0.188s
I20240501 14:01:21.121901 26510 raft_consensus.cc:348] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:21.122351 26510 raft_consensus.cc:729] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Initialized, Role: FOLLOWER
I20240501 14:01:21.123521 26510 consensus_queue.cc:260] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 667, Last appended: 1.667, Last appended by leader: 667, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:21.126834 26510 ts_tablet_manager.cc:1398] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:01:21.131248 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session 5e1c3e3144384f90b47255623d63575b-999759caca8344f2b2ae0cff5e204425 received from {username='slave'} at 127.16.36.133:45659
I20240501 14:01:21.131425 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session 5e1c3e3144384f90b47255623d63575b-999759caca8344f2b2ae0cff5e204425 on tablet 999759caca8344f2b2ae0cff5e204425 with peer 5e1c3e3144384f90b47255623d63575b
I20240501 14:01:21.349325 26491 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=623 overwritten=0 applied=623 ignored=0} inserts{seen=1305 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:21.351740 26491 tablet_bootstrap.cc:492] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: Bootstrap complete.
I20240501 14:01:21.352226 26491 ts_tablet_manager.cc:1367] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: Time spent bootstrapping tablet: real 3.139s user 0.833s sys 0.107s
I20240501 14:01:21.353245 26491 raft_consensus.cc:348] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:01:21.353586 26491 raft_consensus.cc:729] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedba8a189394baca3188f2a5eddbcd4, State: Initialized, Role: FOLLOWER
I20240501 14:01:21.353968 26491 consensus_queue.cc:260] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 623, Last appended: 2.623, Last appended by leader: 623, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:01:21.363085 26491 ts_tablet_manager.cc:1398] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20240501 14:01:21.365908 22302 tablet_copy_service.cc:342] P 474d063159a143fcba0416fa01328bc0: Request end of tablet copy session dedba8a189394baca3188f2a5eddbcd4-dc19025daec44883afec64b3d12c7c4e received from {username='slave'} at 127.16.36.134:56211
I20240501 14:01:21.366117 22302 tablet_copy_service.cc:433] P 474d063159a143fcba0416fa01328bc0: ending tablet copy session dedba8a189394baca3188f2a5eddbcd4-dc19025daec44883afec64b3d12c7c4e on tablet dc19025daec44883afec64b3d12c7c4e with peer dedba8a189394baca3188f2a5eddbcd4
I20240501 14:01:21.500430 23724 raft_consensus.cc:1203] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.658->[1.659-1.695] Dedup: 1.667->[1.668-1.695]
I20240501 14:01:21.533324 23856 raft_consensus.cc:1203] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.615->[2.616-2.673] Dedup: 2.623->[2.624-2.673]
I20240501 14:01:21.960847 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:01:22.067411 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:39481, user_credentials={real_user=slave}} blocked reactor thread for 76206.9us
I20240501 14:01:22.072789 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:22.127084 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:22.130092 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:22.230998 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:22.292109 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:22.365960 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:23.881426 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:23.908612 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:23.909240 22261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:23.909628 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:23.910768 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:23.956864 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:24.069283 23838 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:24.389014 22282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e"
dest_uuid: "474d063159a143fcba0416fa01328bc0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37946
I20240501 14:01:24.389264 22282 raft_consensus.cc:595] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 LEADER]: Received request to transfer leadership
I20240501 14:01:24.391067 26689 raft_consensus.cc:979] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: : Instructing follower dedba8a189394baca3188f2a5eddbcd4 to start an election
I20240501 14:01:24.391216 26689 raft_consensus.cc:1067] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 LEADER]: Signalling peer dedba8a189394baca3188f2a5eddbcd4 to start an election
I20240501 14:01:24.392707 23856 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e"
dest_uuid: "dedba8a189394baca3188f2a5eddbcd4"
from {username='slave'} at 127.16.36.129:42769
I20240501 14:01:24.393064 23856 raft_consensus.cc:482] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:24.393321 23856 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:01:24.394464 22242 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42762: Service unavailable: leader transfer in progress
I20240501 14:01:24.397094 23856 raft_consensus.cc:504] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
W20240501 14:01:24.399828 22242 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42762: Service unavailable: leader transfer in progress
I20240501 14:01:24.401724 22282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e" candidate_uuid: "dedba8a189394baca3188f2a5eddbcd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 724 } } ignore_live_leader: true dest_uuid: "474d063159a143fcba0416fa01328bc0"
I20240501 14:01:24.401919 22282 raft_consensus.cc:3032] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:24.402015 22282 raft_consensus.cc:729] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 474d063159a143fcba0416fa01328bc0, State: Running, Role: LEADER
I20240501 14:01:24.402338 22282 consensus_queue.cc:260] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 724, Committed index: 724, Last appended: 2.724, Last appended by leader: 724, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:01:24.402930 22282 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:01:24.407863 22242 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42762: Illegal state: replica 474d063159a143fcba0416fa01328bc0 is not leader of this config: current role FOLLOWER
I20240501 14:01:24.409881 22282 raft_consensus.cc:2454] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dedba8a189394baca3188f2a5eddbcd4 in term 3.
I20240501 14:01:24.410591 23791 leader_election.cc:302] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 474d063159a143fcba0416fa01328bc0, dedba8a189394baca3188f2a5eddbcd4; no voters:
I20240501 14:01:24.412014 26730 raft_consensus.cc:2781] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:24.425355 23856 leader_election.cc:288] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 3 election: Requested vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963), 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481)
W20240501 14:01:24.435606 22242 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42762: Illegal state: replica 474d063159a143fcba0416fa01328bc0 is not leader of this config: current role FOLLOWER
W20240501 14:01:24.445677 23685 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50396: Illegal state: replica 5e1c3e3144384f90b47255623d63575b is not leader of this config: current role FOLLOWER
I20240501 14:01:24.448348 26730 raft_consensus.cc:686] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 3 LEADER]: Becoming Leader. State: Replica: dedba8a189394baca3188f2a5eddbcd4, State: Running, Role: LEADER
I20240501 14:01:24.457365 26730 consensus_queue.cc:237] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 723, Committed index: 723, Last appended: 2.724, Last appended by leader: 724, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:01:24.490496 23854 raft_consensus.cc:1226] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 3 LEADER]: Rejecting Update request from peer 474d063159a143fcba0416fa01328bc0 for earlier term 2. Current term is 3. Ops: []
I20240501 14:01:24.492842 22137 catalog_manager.cc:5484] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 reported cstate change: term changed from 2 to 3, leader changed from 474d063159a143fcba0416fa01328bc0 (127.16.36.129) to dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134). New cstate: current_term: 3 leader_uuid: "dedba8a189394baca3188f2a5eddbcd4" committed_config { opid_index: 616 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:24.579632 23725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19025daec44883afec64b3d12c7c4e" candidate_uuid: "dedba8a189394baca3188f2a5eddbcd4" candidate_term: 3 candidate_status { last_received { term: 2 index: 724 } } ignore_live_leader: true dest_uuid: "5e1c3e3144384f90b47255623d63575b"
I20240501 14:01:24.579918 23725 raft_consensus.cc:3037] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:24.583017 23725 raft_consensus.cc:2454] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dedba8a189394baca3188f2a5eddbcd4 in term 3.
I20240501 14:01:24.665541 23726 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 3 FOLLOWER]: Refusing update from remote peer dedba8a189394baca3188f2a5eddbcd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 724. Preceding OpId from leader: term: 3 index: 726. (index mismatch)
I20240501 14:01:24.667026 22282 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 3 FOLLOWER]: Refusing update from remote peer dedba8a189394baca3188f2a5eddbcd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 724. Preceding OpId from leader: term: 3 index: 726. (index mismatch)
I20240501 14:01:24.667495 26731 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 725, Last known committed idx: 724, Time since last communication: 0.000s
I20240501 14:01:24.668320 26730 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 725, Last known committed idx: 724, Time since last communication: 0.000s
I20240501 14:01:24.676041 26698 mvcc.cc:205] Tried to move back new op lower bound from 7022887258776297472 to 7022887258062135296. Current Snapshot: MvccSnapshot[applied={T|T < 7022887258776297472 or (T in {7022887258776297472})}]
W20240501 14:01:25.370159 22346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:45725, user_credentials={real_user=slave}} blocked reactor thread for 47813.1us
I20240501 14:01:25.593729 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:25.594327 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:25.603987 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:25.639820 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:25.713266 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:25.784600 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:25.924057 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:01:26.249156 22218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:35555, user_credentials={real_user=slave}} blocked reactor thread for 116477us
I20240501 14:01:26.528537 23854 consensus_queue.cc:237] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 760, Committed index: 760, Last appended: 3.761, Last appended by leader: 724, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:01:26.540158 23724 raft_consensus.cc:1261] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 3 FOLLOWER]: Refusing update from remote peer dedba8a189394baca3188f2a5eddbcd4: Log matching property violated. Preceding OpId in replica: term: 3 index: 761. Preceding OpId from leader: term: 3 index: 762. (index mismatch)
I20240501 14:01:26.542796 26804 consensus_queue.cc:1035] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 762, Last known committed idx: 760, Time since last communication: 0.000s
I20240501 14:01:26.547314 26730 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P dedba8a189394baca3188f2a5eddbcd4 [term 3 LEADER]: Committing config change with OpId 3.762: config changed from index 616 to 762, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:26.548240 23724 raft_consensus.cc:2932] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b [term 3 FOLLOWER]: Committing config change with OpId 3.762: config changed from index 616 to 762, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:26.554492 22139 catalog_manager.cc:5484] T dc19025daec44883afec64b3d12c7c4e P 5e1c3e3144384f90b47255623d63575b reported cstate change: config changed from index 616 to 762, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New cstate: current_term: 3 leader_uuid: "dedba8a189394baca3188f2a5eddbcd4" committed_config { opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } }
I20240501 14:01:26.664258 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet dc19025daec44883afec64b3d12c7c4e with delete_type TABLET_DATA_TOMBSTONED (TS 474d063159a143fcba0416fa01328bc0 not found in new config with opid_index 762) from {username='slave'} at 127.0.0.1:35434
I20240501 14:01:26.665035 26839 tablet_replica.cc:330] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: stopping tablet replica
I20240501 14:01:26.665388 26839 raft_consensus.cc:2229] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:26.665812 26839 pending_rounds.cc:62] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Trying to abort 1 pending ops.
I20240501 14:01:26.666024 26839 pending_rounds.cc:68] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Aborting op as it isn't in flight: id { term: 3 index: 761 } timestamp: 7022887266387177472 op_type: WRITE_OP write_request { tablet_id: "dc19025daec44883afec64b3d12c7c4e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\245s\351\002nNI\236\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c95bcdeed459d3c4" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022887266339586048 authz_token { token_data: "\010\246\230\311\261\006\"3\n\005slave\022*\n aa4393d7657c42b3b15522ad50965d6c\020\001\030\001 \001(\001" signature: "<redacted>""j@\376W\232\315n\302\201\3437/zS\3313\034\t7F\237\240\253\305Jr7\274\246G\305\355hS=wwj\372}Y\002\336\215Q}\327J\025:&\327\372\342\275\2530\212\177}\2156(\206" signing_key_seq_num: 0 } } request_id { client_id: "aee263622d8b46de9226072641598f11" seq_no: 2257 first_incomplete_seq_no: 2257 attempt_no: 0 }
I20240501 14:01:26.666779 26839 raft_consensus.cc:2258] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:26.682924 26839 ts_tablet_manager.cc:1875] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:26.704469 26839 ts_tablet_manager.cc:1888] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.761
I20240501 14:01:26.705060 26839 log.cc:1196] T dc19025daec44883afec64b3d12c7c4e P 474d063159a143fcba0416fa01328bc0: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc19025daec44883afec64b3d12c7c4e
I20240501 14:01:26.708261 22125 catalog_manager.cc:4830] TS 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): tablet dc19025daec44883afec64b3d12c7c4e (table rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]) successfully deleted
I20240501 14:01:27.889164 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:27.938565 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:27.938614 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:27.984405 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:27.985388 23705 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:27.989890 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:28.022835 22392 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:01:28.543799 22411 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "999759caca8344f2b2ae0cff5e204425"
dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36550
I20240501 14:01:28.543982 22411 raft_consensus.cc:595] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Received request to transfer leadership
W20240501 14:01:28.609251 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.622804 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.641450 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.649964 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.656932 22371 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.679333 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.696004 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.711364 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.727339 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.742192 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.767603 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.801375 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.822008 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.850872 22372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.872073 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.891587 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.910856 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
W20240501 14:01:28.933457 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
I20240501 14:01:28.936851 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:28.956759 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:01:28.957202 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Service unavailable: leader transfer in progress
I20240501 14:01:28.957960 26933 raft_consensus.cc:979] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: : Instructing follower 5e1c3e3144384f90b47255623d63575b to start an election
I20240501 14:01:28.958130 26933 raft_consensus.cc:1067] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Signalling peer 5e1c3e3144384f90b47255623d63575b to start an election
I20240501 14:01:28.965719 23726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "999759caca8344f2b2ae0cff5e204425"
dest_uuid: "5e1c3e3144384f90b47255623d63575b"
from {username='slave'} at 127.16.36.130:35465
I20240501 14:01:28.966064 23726 raft_consensus.cc:482] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:28.966213 23726 raft_consensus.cc:3037] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:28.970238 23726 raft_consensus.cc:504] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:28.975522 22411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999759caca8344f2b2ae0cff5e204425" candidate_uuid: "5e1c3e3144384f90b47255623d63575b" candidate_term: 2 candidate_status { last_received { term: 1 index: 803 } } ignore_live_leader: true dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
I20240501 14:01:28.975819 22411 raft_consensus.cc:3032] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:28.975946 22411 raft_consensus.cc:729] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:01:28.976285 22411 consensus_queue.cc:260] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 803, Committed index: 803, Last appended: 1.803, Last appended by leader: 803, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:28.976686 22411 raft_consensus.cc:3037] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:28.982482 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Illegal state: replica a5574cf7d8b54da09d7f589c2c11e97c is not leader of this config: current role FOLLOWER
I20240501 14:01:28.988385 22411 raft_consensus.cc:2454] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c3e3144384f90b47255623d63575b in term 2.
I20240501 14:01:28.995393 23660 leader_election.cc:302] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1c3e3144384f90b47255623d63575b, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:01:28.995904 26930 raft_consensus.cc:2781] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:29.017232 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:29.024405 23726 leader_election.cc:288] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [CANDIDATE]: Term 2 election: Requested vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541), dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555)
I20240501 14:01:29.033632 26930 raft_consensus.cc:686] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c3e3144384f90b47255623d63575b, State: Running, Role: LEADER
I20240501 14:01:29.034157 26930 consensus_queue.cc:237] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 803, Committed index: 803, Last appended: 1.803, Last appended by leader: 803, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:29.038926 22137 catalog_manager.cc:5484] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b reported cstate change: term changed from 1 to 2, leader changed from a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) to 5e1c3e3144384f90b47255623d63575b (127.16.36.133). New cstate: current_term: 2 leader_uuid: "5e1c3e3144384f90b47255623d63575b" committed_config { opid_index: 659 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
W20240501 14:01:29.041594 22373 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38212: Illegal state: replica a5574cf7d8b54da09d7f589c2c11e97c is not leader of this config: current role FOLLOWER
I20240501 14:01:29.118542 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:29.140789 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999759caca8344f2b2ae0cff5e204425" candidate_uuid: "5e1c3e3144384f90b47255623d63575b" candidate_term: 2 candidate_status { last_received { term: 1 index: 803 } } ignore_live_leader: true dest_uuid: "dedba8a189394baca3188f2a5eddbcd4"
I20240501 14:01:29.141187 23858 raft_consensus.cc:3037] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:29.144398 23858 raft_consensus.cc:2454] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c3e3144384f90b47255623d63575b in term 2.
I20240501 14:01:29.168689 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:29.193958 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:29.275949 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:29.295985 23818 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37126: Illegal state: replica dedba8a189394baca3188f2a5eddbcd4 is not leader of this config: current role FOLLOWER
I20240501 14:01:29.337510 23855 raft_consensus.cc:1261] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c3e3144384f90b47255623d63575b: Log matching property violated. Preceding OpId in replica: term: 1 index: 803. Preceding OpId from leader: term: 2 index: 805. (index mismatch)
I20240501 14:01:29.337563 22411 raft_consensus.cc:1261] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Refusing update from remote peer 5e1c3e3144384f90b47255623d63575b: Log matching property violated. Preceding OpId in replica: term: 1 index: 803. Preceding OpId from leader: term: 2 index: 805. (index mismatch)
I20240501 14:01:29.338372 26964 consensus_queue.cc:1035] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 804, Last known committed idx: 803, Time since last communication: 0.000s
I20240501 14:01:29.342003 26964 consensus_queue.cc:1035] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 804, Last known committed idx: 803, Time since last communication: 0.000s
I20240501 14:01:30.460752 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:30.516816 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:30.563601 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:30.608103 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:30.621990 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:30.643309 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:30.870766 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:31.338567 23725 consensus_queue.cc:237] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 840, Committed index: 840, Last appended: 2.840, Last appended by leader: 803, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 841 OBSOLETE_local: false peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } }
I20240501 14:01:31.340649 23858 raft_consensus.cc:1261] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c3e3144384f90b47255623d63575b: Log matching property violated. Preceding OpId in replica: term: 2 index: 840. Preceding OpId from leader: term: 2 index: 841. (index mismatch)
I20240501 14:01:31.344521 27024 consensus_queue.cc:1035] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 841, Last known committed idx: 840, Time since last communication: 0.000s
I20240501 14:01:31.351655 27024 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b [term 2 LEADER]: Committing config change with OpId 2.841: config changed from index 659 to 841, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 841 OBSOLETE_local: false peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:31.353580 23858 raft_consensus.cc:2932] T 999759caca8344f2b2ae0cff5e204425 P dedba8a189394baca3188f2a5eddbcd4 [term 2 FOLLOWER]: Committing config change with OpId 2.841: config changed from index 659 to 841, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 841 OBSOLETE_local: false peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:31.365640 22140 catalog_manager.cc:5484] T 999759caca8344f2b2ae0cff5e204425 P 5e1c3e3144384f90b47255623d63575b reported cstate change: config changed from index 659 to 841, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New cstate: current_term: 2 leader_uuid: "5e1c3e3144384f90b47255623d63575b" committed_config { opid_index: 841 OBSOLETE_local: false peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } }
I20240501 14:01:31.436586 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet 999759caca8344f2b2ae0cff5e204425 with delete_type TABLET_DATA_TOMBSTONED (TS a5574cf7d8b54da09d7f589c2c11e97c not found in new config with opid_index 841) from {username='slave'} at 127.0.0.1:41234
I20240501 14:01:31.439445 27069 tablet_replica.cc:330] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: stopping tablet replica
I20240501 14:01:31.439762 27069 raft_consensus.cc:2229] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:31.440146 27069 raft_consensus.cc:2258] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:31.441282 27069 ts_tablet_manager.cc:1875] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:31.448980 27069 ts_tablet_manager.cc:1888] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.840
I20240501 14:01:31.449208 27069 log.cc:1196] T 999759caca8344f2b2ae0cff5e204425 P a5574cf7d8b54da09d7f589c2c11e97c: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/999759caca8344f2b2ae0cff5e204425
I20240501 14:01:31.451334 22128 catalog_manager.cc:4830] TS a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): tablet 999759caca8344f2b2ae0cff5e204425 (table rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]) successfully deleted
I20240501 14:01:32.590996 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:32.620823 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:32.752826 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:32.773360 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:32.817016 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:32.829502 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:33.017279 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:33.912653 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:33.926764 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:33.927403 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:33.954392 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:33.963749 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:33.990303 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:34.051844 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:34.158612 22218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 62135.2us
I20240501 14:01:36.380690 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:36.384133 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:36.492954 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:36.534969 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:36.540838 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:36.621608 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:36.651062 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:01:38.196328 22476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:39481, user_credentials={real_user=slave}} blocked reactor thread for 64892us
I20240501 14:01:38.207423 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:38.218420 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:38.236250 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:38.276490 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:38.288269 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:38.300302 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:38.307538 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:38.673777 22218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:35555, user_credentials={real_user=slave}} blocked reactor thread for 51598.8us
I20240501 14:01:38.878736 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:38.889432 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:38.893378 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:38.895887 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:38.923372 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:38.923816 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:38.965380 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e95d660c7bd7435fa08604fea52664ad | 127.16.36.190:35677 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.148:42753 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
3b1653dc15114e7c873f9cc1a9a3c0a5 | 127.16.36.131:38305 | HEALTHY | <none> | 4 | 0
474d063159a143fcba0416fa01328bc0 | 127.16.36.129:33963 | HEALTHY | <none> | 4 | 0
5e1c3e3144384f90b47255623d63575b | 127.16.36.133:39481 | HEALTHY | <none> | 2 | 0
90f980f10dfd441289a8ad090e23e200 | 127.16.36.132:45725 | HEALTHY | <none> | 0 | 0
a5574cf7d8b54da09d7f589c2c11e97c | 127.16.36.130:46541 | HEALTHY | <none> | 5 | 0
bcf471e1f6eb42b8a9ae457e6239b89f | 127.16.36.135:41675 | HEALTHY | <none> | 0 | 0
dedba8a189394baca3188f2a5eddbcd4 | 127.16.36.134:35555 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.16.36.129 | experimental | 127.16.36.129:33963
local_ip_for_outbound_sockets | 127.16.36.130 | experimental | 127.16.36.130:46541
local_ip_for_outbound_sockets | 127.16.36.131 | experimental | 127.16.36.131:38305
local_ip_for_outbound_sockets | 127.16.36.132 | experimental | 127.16.36.132:45725
local_ip_for_outbound_sockets | 127.16.36.133 | experimental | 127.16.36.133:39481
local_ip_for_outbound_sockets | 127.16.36.134 | experimental | 127.16.36.134:35555
local_ip_for_outbound_sockets | 127.16.36.135 | experimental | 127.16.36.135:41675
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.36.129:33963
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.36.130:46541
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.36.131:38305
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.36.132:45725
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.36.133:39481
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.36.134:35555
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.36.135:41675
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.148:42753 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:01:39.031510 16530 log_verifier.cc:126] Checking tablet 161faaa672ce4a0e80495369f1060558
I20240501 14:01:39.401780 16530 log_verifier.cc:177] Verified matching terms for 927 ops in tablet 161faaa672ce4a0e80495369f1060558
I20240501 14:01:39.403031 16530 log_verifier.cc:126] Checking tablet 4ebad65f05b645fdbd0844fee2b23dfe
I20240501 14:01:39.726192 16530 log_verifier.cc:177] Verified matching terms for 924 ops in tablet 4ebad65f05b645fdbd0844fee2b23dfe
I20240501 14:01:39.727490 16530 log_verifier.cc:126] Checking tablet 74e03b7d269c40ff9fbf6457611d849b
I20240501 14:01:40.142836 16530 log_verifier.cc:177] Verified matching terms for 1167 ops in tablet 74e03b7d269c40ff9fbf6457611d849b
I20240501 14:01:40.144151 16530 log_verifier.cc:126] Checking tablet 7bcbcadfd9a34282950350f04b7fcce5
I20240501 14:01:40.532019 16530 log_verifier.cc:177] Verified matching terms for 1112 ops in tablet 7bcbcadfd9a34282950350f04b7fcce5
I20240501 14:01:40.533429 16530 log_verifier.cc:126] Checking tablet 83ae513a62f345d1a6eac152a56936d0
I20240501 14:01:40.868642 16530 log_verifier.cc:177] Verified matching terms for 969 ops in tablet 83ae513a62f345d1a6eac152a56936d0
I20240501 14:01:40.869773 16530 log_verifier.cc:126] Checking tablet 999759caca8344f2b2ae0cff5e204425
I20240501 14:01:41.191674 16530 log_verifier.cc:177] Verified matching terms for 972 ops in tablet 999759caca8344f2b2ae0cff5e204425
I20240501 14:01:41.192823 16530 log_verifier.cc:126] Checking tablet a40334656b4b4333a13956fec5a1053a
I20240501 14:01:41.565318 16530 log_verifier.cc:177] Verified matching terms for 923 ops in tablet a40334656b4b4333a13956fec5a1053a
I20240501 14:01:41.566543 16530 log_verifier.cc:126] Checking tablet b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:01:42.052742 16530 log_verifier.cc:177] Verified matching terms for 1093 ops in tablet b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:01:42.054028 16530 log_verifier.cc:126] Checking tablet c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:01:42.488085 16530 log_verifier.cc:177] Verified matching terms for 967 ops in tablet c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:01:42.489276 16530 log_verifier.cc:126] Checking tablet c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:01:42.917529 16530 log_verifier.cc:177] Verified matching terms for 1095 ops in tablet c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:01:42.919950 16530 log_verifier.cc:126] Checking tablet ce58c7fa6c734d3fb453d132540f933c
I20240501 14:01:42.982580 16530 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ce58c7fa6c734d3fb453d132540f933c
I20240501 14:01:42.982726 16530 log_verifier.cc:126] Checking tablet d4716a6313a541b981e5d27b01657956
I20240501 14:01:43.381909 16530 log_verifier.cc:177] Verified matching terms for 982 ops in tablet d4716a6313a541b981e5d27b01657956
I20240501 14:01:43.383157 16530 log_verifier.cc:126] Checking tablet dc19025daec44883afec64b3d12c7c4e
I20240501 14:01:43.750967 16530 log_verifier.cc:177] Verified matching terms for 952 ops in tablet dc19025daec44883afec64b3d12c7c4e
I20240501 14:01:43.752394 16530 log_verifier.cc:126] Checking tablet dd5f6e034ac14cd2864d894fe5b00fd9
I20240501 14:01:44.135012 16530 log_verifier.cc:177] Verified matching terms for 945 ops in tablet dd5f6e034ac14cd2864d894fe5b00fd9
I20240501 14:01:44.136201 16530 log_verifier.cc:126] Checking tablet e6ac874a7f084076bf03ed4167f967e0
I20240501 14:01:44.560256 16530 log_verifier.cc:177] Verified matching terms for 1023 ops in tablet e6ac874a7f084076bf03ed4167f967e0
I20240501 14:01:44.561476 16530 log_verifier.cc:126] Checking tablet fe968af0ff994a9093beca7a614e77b7
I20240501 14:01:45.002151 16530 log_verifier.cc:177] Verified matching terms for 1051 ops in tablet fe968af0ff994a9093beca7a614e77b7
I20240501 14:01:45.018250 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.136:0
--local_ip_for_outbound_sockets=127.16.36.136
--webserver_interface=127.16.36.136
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--builtin_ntp_servers=127.16.36.148:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:01:45.547821 27397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:45.548189 27397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:45.548349 27397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:45.558187 27397 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:01:45.558362 27397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:45.558581 27397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.136
I20240501 14:01:45.569919 27397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.136
--webserver_port=0
--tserver_master_addrs=127.16.36.190:35677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.136
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:45.571431 27397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:45.573403 27397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:45.588510 27406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:45.588917 27403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:45.589491 27404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:45.590210 27397 server_base.cc:1034] running on GCE node
I20240501 14:01:45.590896 27397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:45.591907 27397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:45.593139 27397 hybrid_clock.cc:648] HybridClock initialized: now 1714572105593077 us; error 42 us; skew 500 ppm
I20240501 14:01:45.593493 27397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:45.596439 27397 webserver.cc:460] Webserver started at http://127.16.36.136:38229/ using document root <none> and password file <none>
I20240501 14:01:45.597306 27397 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:45.597442 27397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:45.597828 27397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:45.601491 27397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "13c50a9f8b674ac19f095e891401aff9"
format_stamp: "Formatted at 2024-05-01 14:01:45 on dist-test-slave-9q9t"
I20240501 14:01:45.602509 27397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "13c50a9f8b674ac19f095e891401aff9"
format_stamp: "Formatted at 2024-05-01 14:01:45 on dist-test-slave-9q9t"
I20240501 14:01:45.608381 27397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:01:45.612684 27412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.614151 27397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:45.614332 27397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "13c50a9f8b674ac19f095e891401aff9"
format_stamp: "Formatted at 2024-05-01 14:01:45 on dist-test-slave-9q9t"
I20240501 14:01:45.614596 27397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:45.643301 27397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:45.644430 27397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:45.644780 27397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:45.646031 27397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:45.647975 27397 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:45.648093 27397 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.648243 27397 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:45.648351 27397 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.705539 27397 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.136:46165
I20240501 14:01:45.705616 27525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.136:46165 every 8 connection(s)
I20240501 14:01:45.707276 27397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:01:45.713074 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 27397
I20240501 14:01:45.714929 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:01:45.724637 27526 heartbeater.cc:347] Connected to a master server at 127.16.36.190:35677
I20240501 14:01:45.725154 27526 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:45.726512 27526 heartbeater.cc:511] Master 127.16.36.190:35677 requested a full tablet report, sending...
I20240501 14:01:45.728653 22136 ts_manager.cc:194] Registered new tserver with Master: 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165)
I20240501 14:01:45.729908 22136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.136:54857
I20240501 14:01:46.403584 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:46.427681 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:46.454339 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:46.461683 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:46.474761 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:46.524665 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:46.528007 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:46.538511 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:46.735457 27526 heartbeater.cc:503] Master 127.16.36.190:35677 was elected leader, sending a full tablet report...
I20240501 14:01:46.744982 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:46.767431 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:46.785646 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:46.790594 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:46.791846 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:46.807901 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:46.853336 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:46.853350 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:46.959239 22541 consensus_queue.cc:237] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1095, Committed index: 1095, Last appended: 1.1095, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:46.961982 23855 raft_consensus.cc:1261] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1095. Preceding OpId from leader: term: 1 index: 1096. (index mismatch)
I20240501 14:01:46.962576 27193 consensus_queue.cc:1035] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1096, Last known committed idx: 1095, Time since last communication: 0.000s
I20240501 14:01:46.965591 27634 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.1096: config changed from index 552 to 1096, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:46.966410 23855 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Committing config change with OpId 1.1096: config changed from index 552 to 1096, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
W20240501 14:01:46.969213 22478 consensus_peers.cc:485] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c40fd7b8bfd04d8ca67a9acf40f74445. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:46.972576 22137 catalog_manager.cc:5484] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 reported cstate change: config changed from index 552 to 1096, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.011828 22411 consensus_queue.cc:237] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 967, Committed index: 967, Last appended: 1.967, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:47.014559 23588 raft_consensus.cc:1261] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Refusing update from remote peer a5574cf7d8b54da09d7f589c2c11e97c: Log matching property violated. Preceding OpId in replica: term: 1 index: 967. Preceding OpId from leader: term: 1 index: 968. (index mismatch)
I20240501 14:01:47.015308 27658 consensus_queue.cc:1035] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 968, Last known committed idx: 967, Time since last communication: 0.000s
I20240501 14:01:47.018121 26764 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Committing config change with OpId 1.968: config changed from index 371 to 968, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
W20240501 14:01:47.020946 22348 consensus_peers.cc:485] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c09f45be8c064ecdbdc78d5ce8dbe8bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.022327 23588 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Committing config change with OpId 1.968: config changed from index 371 to 968, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.026768 22134 catalog_manager.cc:5484] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c reported cstate change: config changed from index 371 to 968, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" committed_config { opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.066519 22541 consensus_queue.cc:237] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1093, Committed index: 1093, Last appended: 1.1093, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:47.068964 23725 raft_consensus.cc:1261] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1093. Preceding OpId from leader: term: 1 index: 1094. (index mismatch)
W20240501 14:01:47.069208 22478 consensus_peers.cc:485] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b28e7ed3a23243ee98327f60b2ebdbd4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.069548 27637 consensus_queue.cc:1035] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1094, Last known committed idx: 1093, Time since last communication: 0.000s
I20240501 14:01:47.072178 27634 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.1094: config changed from index 546 to 1094, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.072875 23725 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Committing config change with OpId 1.1094: config changed from index 546 to 1094, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.077116 22137 catalog_manager.cc:5484] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b reported cstate change: config changed from index 546 to 1094, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.110021 22541 consensus_queue.cc:237] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 924, Committed index: 924, Last appended: 1.924, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:47.112329 22281 raft_consensus.cc:1261] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 924. Preceding OpId from leader: term: 1 index: 925. (index mismatch)
W20240501 14:01:47.112527 22478 consensus_peers.cc:485] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ebad65f05b645fdbd0844fee2b23dfe. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:47.112913 27634 consensus_queue.cc:1035] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 925, Last known committed idx: 924, Time since last communication: 0.000s
I20240501 14:01:47.115506 27637 raft_consensus.cc:2932] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.925: config changed from index -1 to 925, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.116139 22281 raft_consensus.cc:2932] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Committing config change with OpId 1.925: config changed from index -1 to 925, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New config: { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.121318 22134 catalog_manager.cc:5484] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index -1 to 925, VOTER 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:47.247327 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.268942 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.279947 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:47.283216 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.284690 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:47.291501 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:47.293840 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:47.309540 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:47.506912 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.507380 27768 ts_tablet_manager.cc:902] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: Initiating tablet copy from peer 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:01:47.530829 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:47.545089 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:47.548019 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:47.560429 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.566780 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.573042 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:47.593523 27768 tablet_copy_client.cc:308] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:38305
I20240501 14:01:47.595438 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:47.622800 27777 ts_tablet_manager.cc:902] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: Initiating tablet copy from peer 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:01:47.647258 22563 tablet_copy_service.cc:140] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Received BeginTabletCopySession request for tablet c40fd7b8bfd04d8ca67a9acf40f74445 from peer 13c50a9f8b674ac19f095e891401aff9 ({username='slave'} at 127.16.36.136:51945)
I20240501 14:01:47.647472 22563 tablet_copy_service.cc:161] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Beginning new tablet copy session on tablet c40fd7b8bfd04d8ca67a9acf40f74445 from peer 13c50a9f8b674ac19f095e891401aff9 at {username='slave'} at 127.16.36.136:51945: session id = 13c50a9f8b674ac19f095e891401aff9-c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:01:47.651189 22563 tablet_copy_source_session.cc:205] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.651226 27777 tablet_copy_client.cc:308] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:38305
I20240501 14:01:47.652150 22562 tablet_copy_service.cc:140] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Received BeginTabletCopySession request for tablet b28e7ed3a23243ee98327f60b2ebdbd4 from peer 13c50a9f8b674ac19f095e891401aff9 ({username='slave'} at 127.16.36.136:51945)
I20240501 14:01:47.652390 22562 tablet_copy_service.cc:161] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Beginning new tablet copy session on tablet b28e7ed3a23243ee98327f60b2ebdbd4 from peer 13c50a9f8b674ac19f095e891401aff9 at {username='slave'} at 127.16.36.136:51945: session id = 13c50a9f8b674ac19f095e891401aff9-b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:01:47.653535 27779 ts_tablet_manager.cc:902] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: Initiating tablet copy from peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
I20240501 14:01:47.655611 22562 tablet_copy_source_session.cc:205] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.660341 27768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40fd7b8bfd04d8ca67a9acf40f74445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.665659 27777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28e7ed3a23243ee98327f60b2ebdbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.686969 27768 tablet_copy_client.cc:783] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.689293 27768 tablet_copy_client.cc:647] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.688901 27777 tablet_copy_client.cc:783] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.689751 27777 tablet_copy_client.cc:647] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.704644 27777 tablet_copy_client.cc:523] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:47.708333 27768 tablet_copy_client.cc:523] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:47.715592 27777 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: Bootstrap starting.
I20240501 14:01:47.718005 27786 ts_tablet_manager.cc:902] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: Initiating tablet copy from peer 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305)
I20240501 14:01:47.726023 27768 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: Bootstrap starting.
I20240501 14:01:47.759696 27779 tablet_copy_client.cc:308] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:46541
I20240501 14:01:47.772678 22433 tablet_copy_service.cc:140] P a5574cf7d8b54da09d7f589c2c11e97c: Received BeginTabletCopySession request for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf from peer 13c50a9f8b674ac19f095e891401aff9 ({username='slave'} at 127.16.36.136:56103)
I20240501 14:01:47.772882 22433 tablet_copy_service.cc:161] P a5574cf7d8b54da09d7f589c2c11e97c: Beginning new tablet copy session on tablet c09f45be8c064ecdbdc78d5ce8dbe8bf from peer 13c50a9f8b674ac19f095e891401aff9 at {username='slave'} at 127.16.36.136:56103: session id = 13c50a9f8b674ac19f095e891401aff9-c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:01:47.776787 22433 tablet_copy_source_session.cc:205] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.779538 27779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.795753 27779 tablet_copy_client.cc:783] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.796396 27779 tablet_copy_client.cc:647] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.797734 27786 tablet_copy_client.cc:308] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:38305
I20240501 14:01:47.798837 22562 tablet_copy_service.cc:140] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Received BeginTabletCopySession request for tablet 4ebad65f05b645fdbd0844fee2b23dfe from peer 13c50a9f8b674ac19f095e891401aff9 ({username='slave'} at 127.16.36.136:51945)
I20240501 14:01:47.799063 22562 tablet_copy_service.cc:161] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Beginning new tablet copy session on tablet 4ebad65f05b645fdbd0844fee2b23dfe from peer 13c50a9f8b674ac19f095e891401aff9 at {username='slave'} at 127.16.36.136:51945: session id = 13c50a9f8b674ac19f095e891401aff9-4ebad65f05b645fdbd0844fee2b23dfe
I20240501 14:01:47.802600 22562 tablet_copy_source_session.cc:205] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:47.807083 27779 tablet_copy_client.cc:523] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:47.811679 27779 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: Bootstrap starting.
I20240501 14:01:47.812189 27786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebad65f05b645fdbd0844fee2b23dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:47.841060 27786 tablet_copy_client.cc:783] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:47.841439 27786 tablet_copy_client.cc:647] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:47.851965 27786 tablet_copy_client.cc:523] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:47.868386 27786 tablet_bootstrap.cc:492] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: Bootstrap starting.
I20240501 14:01:48.056990 27779 log.cc:824] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:48.186542 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:48.210642 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:48.232230 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:48.273715 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:48.274755 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:48.320276 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:48.321295 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:48.323477 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:48.914153 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:48.932600 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:48.951102 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:48.951869 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:48.978135 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:49.008673 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:49.097218 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:49.118386 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:49.772297 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:49.787305 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:01:49.810370 22478 consensus_peers.cc:485] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:49.836263 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:49.870443 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:49.870975 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:49.876963 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:49.892683 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:49.932374 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:01:50.071203 22478 consensus_peers.cc:485] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:50.093477 22478 consensus_peers.cc:485] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:50.157624 22348 consensus_peers.cc:485] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c -> Peer 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136:46165): Couldn't send request to peer 13c50a9f8b674ac19f095e891401aff9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:50.385426 27786 tablet_bootstrap.cc:492] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=925 overwritten=0 applied=925 ignored=0} inserts{seen=1556 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:50.386255 27786 tablet_bootstrap.cc:492] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: Bootstrap complete.
I20240501 14:01:50.386973 27786 ts_tablet_manager.cc:1367] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: Time spent bootstrapping tablet: real 2.519s user 1.098s sys 0.165s
I20240501 14:01:50.401741 27786 raft_consensus.cc:348] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.402395 27786 raft_consensus.cc:729] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c50a9f8b674ac19f095e891401aff9, State: Initialized, Role: FOLLOWER
I20240501 14:01:50.403349 27786 consensus_queue.cc:260] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 925, Last appended: 1.925, Last appended by leader: 925, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 925 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.414487 27786 ts_tablet_manager.cc:1398] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9: Time spent starting tablet: real 0.027s user 0.013s sys 0.005s
I20240501 14:01:50.416160 22562 tablet_copy_service.cc:342] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Request end of tablet copy session 13c50a9f8b674ac19f095e891401aff9-4ebad65f05b645fdbd0844fee2b23dfe received from {username='slave'} at 127.16.36.136:51945
I20240501 14:01:50.416368 22562 tablet_copy_service.cc:433] P 3b1653dc15114e7c873f9cc1a9a3c0a5: ending tablet copy session 13c50a9f8b674ac19f095e891401aff9-4ebad65f05b645fdbd0844fee2b23dfe on tablet 4ebad65f05b645fdbd0844fee2b23dfe with peer 13c50a9f8b674ac19f095e891401aff9
I20240501 14:01:50.441365 27779 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=968 overwritten=0 applied=968 ignored=0} inserts{seen=1638 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:50.442098 27779 tablet_bootstrap.cc:492] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: Bootstrap complete.
I20240501 14:01:50.442509 27779 ts_tablet_manager.cc:1367] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: Time spent bootstrapping tablet: real 2.631s user 1.151s sys 0.204s
I20240501 14:01:50.444850 27779 raft_consensus.cc:348] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.445096 27779 raft_consensus.cc:729] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c50a9f8b674ac19f095e891401aff9, State: Initialized, Role: FOLLOWER
I20240501 14:01:50.445477 27779 consensus_queue.cc:260] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 968, Last appended: 1.968, Last appended by leader: 968, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.446183 27779 ts_tablet_manager.cc:1398] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:50.453213 22433 tablet_copy_service.cc:342] P a5574cf7d8b54da09d7f589c2c11e97c: Request end of tablet copy session 13c50a9f8b674ac19f095e891401aff9-c09f45be8c064ecdbdc78d5ce8dbe8bf received from {username='slave'} at 127.16.36.136:56103
I20240501 14:01:50.453424 22433 tablet_copy_service.cc:433] P a5574cf7d8b54da09d7f589c2c11e97c: ending tablet copy session 13c50a9f8b674ac19f095e891401aff9-c09f45be8c064ecdbdc78d5ce8dbe8bf on tablet c09f45be8c064ecdbdc78d5ce8dbe8bf with peer 13c50a9f8b674ac19f095e891401aff9
I20240501 14:01:50.465842 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:50.496181 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:50.527628 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:50.567448 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:50.572141 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:50.647841 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:50.659036 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:50.666363 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:50.674037 27480 raft_consensus.cc:1203] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.924->[1.925-1.925] Dedup: 1.925->[]
I20240501 14:01:50.685691 27768 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1096 overwritten=0 applied=1096 ignored=0} inserts{seen=1725 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:50.686818 27768 tablet_bootstrap.cc:492] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: Bootstrap complete.
I20240501 14:01:50.687669 27768 ts_tablet_manager.cc:1367] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: Time spent bootstrapping tablet: real 2.962s user 1.302s sys 0.214s
I20240501 14:01:50.688531 27768 raft_consensus.cc:348] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.688786 27768 raft_consensus.cc:729] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c50a9f8b674ac19f095e891401aff9, State: Initialized, Role: FOLLOWER
I20240501 14:01:50.689110 27768 consensus_queue.cc:260] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1096, Last appended: 1.1096, Last appended by leader: 1096, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.695341 27768 ts_tablet_manager.cc:1398] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:01:50.698251 22562 tablet_copy_service.cc:342] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Request end of tablet copy session 13c50a9f8b674ac19f095e891401aff9-c40fd7b8bfd04d8ca67a9acf40f74445 received from {username='slave'} at 127.16.36.136:51945
I20240501 14:01:50.698431 22562 tablet_copy_service.cc:433] P 3b1653dc15114e7c873f9cc1a9a3c0a5: ending tablet copy session 13c50a9f8b674ac19f095e891401aff9-c40fd7b8bfd04d8ca67a9acf40f74445 on tablet c40fd7b8bfd04d8ca67a9acf40f74445 with peer 13c50a9f8b674ac19f095e891401aff9
I20240501 14:01:50.715296 27480 raft_consensus.cc:1203] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.967->[1.968-1.968] Dedup: 1.968->[]
I20240501 14:01:50.774729 27777 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1094 overwritten=0 applied=1094 ignored=0} inserts{seen=1764 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:50.775194 27777 tablet_bootstrap.cc:492] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: Bootstrap complete.
I20240501 14:01:50.775552 27777 ts_tablet_manager.cc:1367] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: Time spent bootstrapping tablet: real 3.061s user 1.301s sys 0.195s
I20240501 14:01:50.776190 27777 raft_consensus.cc:348] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.776389 27777 raft_consensus.cc:729] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c50a9f8b674ac19f095e891401aff9, State: Initialized, Role: FOLLOWER
I20240501 14:01:50.776615 27777 consensus_queue.cc:260] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1094, Last appended: 1.1094, Last appended by leader: 1094, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:50.777112 27777 ts_tablet_manager.cc:1398] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:50.777969 22562 tablet_copy_service.cc:342] P 3b1653dc15114e7c873f9cc1a9a3c0a5: Request end of tablet copy session 13c50a9f8b674ac19f095e891401aff9-b28e7ed3a23243ee98327f60b2ebdbd4 received from {username='slave'} at 127.16.36.136:51945
I20240501 14:01:50.778112 22562 tablet_copy_service.cc:433] P 3b1653dc15114e7c873f9cc1a9a3c0a5: ending tablet copy session 13c50a9f8b674ac19f095e891401aff9-b28e7ed3a23243ee98327f60b2ebdbd4 on tablet b28e7ed3a23243ee98327f60b2ebdbd4 with peer 13c50a9f8b674ac19f095e891401aff9
I20240501 14:01:50.793503 27480 raft_consensus.cc:1203] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1095->[1.1096-1.1096] Dedup: 1.1096->[]
I20240501 14:01:51.060577 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.083837 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:51.089712 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:51.091248 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:51.104175 27480 raft_consensus.cc:1203] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1093->[1.1094-1.1094] Dedup: 1.1094->[]
I20240501 14:01:51.111012 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.111512 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:51.112212 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:51.140036 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:01:51.209272 22479 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:41675, user_credentials={real_user=slave}} blocked reactor thread for 75027.1us
I20240501 14:01:51.286226 22541 consensus_queue.cc:237] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 925, Committed index: 925, Last appended: 1.925, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 926 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:51.288620 27480 raft_consensus.cc:1261] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 925. Preceding OpId from leader: term: 1 index: 926. (index mismatch)
I20240501 14:01:51.289304 27836 consensus_queue.cc:1035] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 926, Last known committed idx: 925, Time since last communication: 0.000s
I20240501 14:01:51.293165 27836 raft_consensus.cc:2932] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.926: config changed from index 925 to 926, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 926 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:51.294710 27480 raft_consensus.cc:2932] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Committing config change with OpId 1.926: config changed from index 925 to 926, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New config: { opid_index: 926 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:51.299239 22140 catalog_manager.cc:5484] T 4ebad65f05b645fdbd0844fee2b23dfe P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index 925 to 926, VOTER 474d063159a143fcba0416fa01328bc0 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 926 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:51.306545 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ebad65f05b645fdbd0844fee2b23dfe with delete_type TABLET_DATA_TOMBSTONED (TS 474d063159a143fcba0416fa01328bc0 not found in new config with opid_index 926) from {username='slave'} at 127.0.0.1:35434
I20240501 14:01:51.307294 28034 tablet_replica.cc:330] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: stopping tablet replica
I20240501 14:01:51.307488 28034 raft_consensus.cc:2229] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:51.307736 28034 raft_consensus.cc:2258] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:51.309562 28034 ts_tablet_manager.cc:1875] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:51.315255 28034 ts_tablet_manager.cc:1888] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.925
I20240501 14:01:51.315375 28034 log.cc:1196] T 4ebad65f05b645fdbd0844fee2b23dfe P 474d063159a143fcba0416fa01328bc0: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ebad65f05b645fdbd0844fee2b23dfe
I20240501 14:01:51.316246 22125 catalog_manager.cc:4830] TS 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): tablet 4ebad65f05b645fdbd0844fee2b23dfe (table rebalance_test_table_0 [id=ea4388daf4b24571b5cb400f715debba]) successfully deleted
I20240501 14:01:51.416213 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.457727 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:51.461457 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:51.473618 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:51.474072 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:51.480978 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.486413 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:51.491245 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.609289 22541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b28e7ed3a23243ee98327f60b2ebdbd4"
dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47396
I20240501 14:01:51.609526 22541 raft_consensus.cc:595] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:51.733515 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.742501 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:51.748669 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:51.771431 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.792304 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:51.793900 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:51.795099 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:51.805639 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:52.046923 22523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:52.047806 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:52.048251 27460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:52.094602 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:52.094799 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:52.101950 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:52.122926 28149 raft_consensus.cc:979] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: : Instructing follower 13c50a9f8b674ac19f095e891401aff9 to start an election
I20240501 14:01:52.123337 27836 raft_consensus.cc:1067] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Signalling peer 13c50a9f8b674ac19f095e891401aff9 to start an election
I20240501 14:01:52.124269 27480 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b28e7ed3a23243ee98327f60b2ebdbd4"
dest_uuid: "13c50a9f8b674ac19f095e891401aff9"
from {username='slave'} at 127.16.36.131:57007
I20240501 14:01:52.124588 27480 raft_consensus.cc:482] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:52.124796 27480 raft_consensus.cc:3037] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:52.126323 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:52.137408 27480 raft_consensus.cc:504] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:52.147334 27480 leader_election.cc:288] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [CANDIDATE]: Term 2 election: Requested vote from peers 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305), 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481)
I20240501 14:01:52.148226 22541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28e7ed3a23243ee98327f60b2ebdbd4" candidate_uuid: "13c50a9f8b674ac19f095e891401aff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1094 } } ignore_live_leader: true dest_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5"
I20240501 14:01:52.148487 22541 raft_consensus.cc:3032] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:52.148574 22541 raft_consensus.cc:729] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b1653dc15114e7c873f9cc1a9a3c0a5, State: Running, Role: LEADER
I20240501 14:01:52.148902 22541 consensus_queue.cc:260] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1094, Committed index: 1094, Last appended: 1.1094, Last appended by leader: 1094, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:52.149261 22541 raft_consensus.cc:3037] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:52.152609 22541 raft_consensus.cc:2454] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c50a9f8b674ac19f095e891401aff9 in term 2.
I20240501 14:01:52.153286 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:52.163385 27413 leader_election.cc:302] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c50a9f8b674ac19f095e891401aff9, 3b1653dc15114e7c873f9cc1a9a3c0a5; no voters:
I20240501 14:01:52.187243 28151 raft_consensus.cc:2781] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:52.194607 23726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28e7ed3a23243ee98327f60b2ebdbd4" candidate_uuid: "13c50a9f8b674ac19f095e891401aff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1094 } } ignore_live_leader: true dest_uuid: "5e1c3e3144384f90b47255623d63575b"
I20240501 14:01:52.194883 23726 raft_consensus.cc:3037] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:52.197822 23726 raft_consensus.cc:2454] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c50a9f8b674ac19f095e891401aff9 in term 2.
I20240501 14:01:52.198014 28151 raft_consensus.cc:686] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 2 LEADER]: Becoming Leader. State: Replica: 13c50a9f8b674ac19f095e891401aff9, State: Running, Role: LEADER
I20240501 14:01:52.199337 28151 consensus_queue.cc:237] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1094, Committed index: 1094, Last appended: 1.1094, Last appended by leader: 1094, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:52.213845 22132 catalog_manager.cc:5484] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 reported cstate change: term changed from 1 to 2, leader changed from 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) to 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136). New cstate: current_term: 2 leader_uuid: "13c50a9f8b674ac19f095e891401aff9" committed_config { opid_index: 1094 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:52.283463 22541 consensus_queue.cc:237] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1096, Committed index: 1096, Last appended: 1.1096, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1097 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:52.285864 27480 raft_consensus.cc:1261] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Refusing update from remote peer 3b1653dc15114e7c873f9cc1a9a3c0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1096. Preceding OpId from leader: term: 1 index: 1097. (index mismatch)
I20240501 14:01:52.286576 28149 consensus_queue.cc:1035] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1097, Last known committed idx: 1096, Time since last communication: 0.000s
I20240501 14:01:52.289734 27836 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 1 LEADER]: Committing config change with OpId 1.1097: config changed from index 1096 to 1097, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) evicted. New config: { opid_index: 1097 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:52.290503 27480 raft_consensus.cc:2932] T c40fd7b8bfd04d8ca67a9acf40f74445 P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Committing config change with OpId 1.1097: config changed from index 1096 to 1097, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) evicted. New config: { opid_index: 1097 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:52.295938 22132 catalog_manager.cc:5484] T c40fd7b8bfd04d8ca67a9acf40f74445 P 3b1653dc15114e7c873f9cc1a9a3c0a5 reported cstate change: config changed from index 1096 to 1097, VOTER dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134) evicted. New cstate: current_term: 1 leader_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" committed_config { opid_index: 1097 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:52.306903 23838 tablet_service.cc:1514] Processing DeleteTablet for tablet c40fd7b8bfd04d8ca67a9acf40f74445 with delete_type TABLET_DATA_TOMBSTONED (TS dedba8a189394baca3188f2a5eddbcd4 not found in new config with opid_index 1097) from {username='slave'} at 127.0.0.1:41606
I20240501 14:01:52.307833 28180 tablet_replica.cc:330] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: stopping tablet replica
I20240501 14:01:52.308377 28180 raft_consensus.cc:2229] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:52.308806 28180 raft_consensus.cc:2258] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:52.311110 28180 ts_tablet_manager.cc:1875] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:52.317833 28180 ts_tablet_manager.cc:1888] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1096
I20240501 14:01:52.318017 28180 log.cc:1196] T c40fd7b8bfd04d8ca67a9acf40f74445 P dedba8a189394baca3188f2a5eddbcd4: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/wals/c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:01:52.319200 22128 catalog_manager.cc:4830] TS dedba8a189394baca3188f2a5eddbcd4 (127.16.36.134:35555): tablet c40fd7b8bfd04d8ca67a9acf40f74445 (table rebalance_test_table_3 [id=e6e5f3c2f86d49159c05269d49a6317a]) successfully deleted
I20240501 14:01:52.401844 27460 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:52.429581 22523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:52.431586 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:52.434553 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:52.444759 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:52.447891 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:52.449268 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:52.488266 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:52.584487 22541 raft_consensus.cc:1261] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Refusing update from remote peer 13c50a9f8b674ac19f095e891401aff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1094. Preceding OpId from leader: term: 2 index: 1095. (index mismatch)
I20240501 14:01:52.585703 28151 consensus_queue.cc:1035] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1095, Last known committed idx: 1094, Time since last communication: 0.000s
I20240501 14:01:52.595276 22411 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c09f45be8c064ecdbdc78d5ce8dbe8bf"
dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44838
I20240501 14:01:52.595571 22411 raft_consensus.cc:595] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:52.598945 23726 raft_consensus.cc:1261] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Refusing update from remote peer 13c50a9f8b674ac19f095e891401aff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1094. Preceding OpId from leader: term: 2 index: 1095. (index mismatch)
I20240501 14:01:52.599634 28152 consensus_queue.cc:1035] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1095, Last known committed idx: 1094, Time since last communication: 0.000s
I20240501 14:01:52.692818 27460 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:52.714007 22523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:52.716120 22392 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:01:52.720679 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:52.723189 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:52.724699 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:52.726895 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:52.731313 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:52.756966 27658 raft_consensus.cc:979] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: : Instructing follower 13c50a9f8b674ac19f095e891401aff9 to start an election
I20240501 14:01:52.757181 27658 raft_consensus.cc:1067] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Signalling peer 13c50a9f8b674ac19f095e891401aff9 to start an election
I20240501 14:01:52.757967 27480 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c09f45be8c064ecdbdc78d5ce8dbe8bf"
dest_uuid: "13c50a9f8b674ac19f095e891401aff9"
from {username='slave'} at 127.16.36.130:35579
I20240501 14:01:52.758261 27480 raft_consensus.cc:482] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:52.758416 27480 raft_consensus.cc:3037] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:52.761672 27480 raft_consensus.cc:504] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:52.763197 22411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09f45be8c064ecdbdc78d5ce8dbe8bf" candidate_uuid: "13c50a9f8b674ac19f095e891401aff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 968 } } ignore_live_leader: true dest_uuid: "a5574cf7d8b54da09d7f589c2c11e97c"
I20240501 14:01:52.763474 22411 raft_consensus.cc:3032] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:52.763602 22411 raft_consensus.cc:729] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5574cf7d8b54da09d7f589c2c11e97c, State: Running, Role: LEADER
I20240501 14:01:52.763900 22411 consensus_queue.cc:260] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 968, Committed index: 968, Last appended: 1.968, Last appended by leader: 968, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:52.764263 22411 raft_consensus.cc:3037] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:52.765528 27480 leader_election.cc:288] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [CANDIDATE]: Term 2 election: Requested vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541), 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725)
I20240501 14:01:52.768151 22411 raft_consensus.cc:2454] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c50a9f8b674ac19f095e891401aff9 in term 2.
I20240501 14:01:52.768679 27416 leader_election.cc:302] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c50a9f8b674ac19f095e891401aff9, a5574cf7d8b54da09d7f589c2c11e97c; no voters:
I20240501 14:01:52.769243 28152 raft_consensus.cc:2781] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:52.769454 28152 raft_consensus.cc:686] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 2 LEADER]: Becoming Leader. State: Replica: 13c50a9f8b674ac19f095e891401aff9, State: Running, Role: LEADER
I20240501 14:01:52.769810 28152 consensus_queue.cc:237] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 968, Committed index: 968, Last appended: 1.968, Last appended by leader: 968, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:52.817117 22137 catalog_manager.cc:5484] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 reported cstate change: term changed from 1 to 2, leader changed from a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) to 13c50a9f8b674ac19f095e891401aff9 (127.16.36.136). New cstate: current_term: 2 leader_uuid: "13c50a9f8b674ac19f095e891401aff9" committed_config { opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:52.831110 23588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09f45be8c064ecdbdc78d5ce8dbe8bf" candidate_uuid: "13c50a9f8b674ac19f095e891401aff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 968 } } ignore_live_leader: true dest_uuid: "90f980f10dfd441289a8ad090e23e200"
I20240501 14:01:52.831487 23588 raft_consensus.cc:3037] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:52.835618 23588 raft_consensus.cc:2454] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c50a9f8b674ac19f095e891401aff9 in term 2.
I20240501 14:01:53.203675 22411 raft_consensus.cc:1261] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Refusing update from remote peer 13c50a9f8b674ac19f095e891401aff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 968. Preceding OpId from leader: term: 2 index: 969. (index mismatch)
I20240501 14:01:53.204732 28152 consensus_queue.cc:1035] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 969, Last known committed idx: 968, Time since last communication: 0.000s
I20240501 14:01:53.227339 23589 raft_consensus.cc:1261] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Refusing update from remote peer 13c50a9f8b674ac19f095e891401aff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 968. Preceding OpId from leader: term: 2 index: 969. (index mismatch)
I20240501 14:01:53.229228 28152 consensus_queue.cc:1035] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 969, Last known committed idx: 968, Time since last communication: 0.000s
I20240501 14:01:53.272567 27460 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:53.297264 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:53.310282 22523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:53.316794 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:53.359263 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:53.359731 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:53.360821 22392 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:53.393298 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:53.467351 27480 consensus_queue.cc:237] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1095, Committed index: 1095, Last appended: 2.1095, Last appended by leader: 1094, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:53.469139 23726 raft_consensus.cc:1261] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Refusing update from remote peer 13c50a9f8b674ac19f095e891401aff9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1095. Preceding OpId from leader: term: 2 index: 1096. (index mismatch)
I20240501 14:01:53.469785 28151 consensus_queue.cc:1035] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1096, Last known committed idx: 1095, Time since last communication: 0.000s
I20240501 14:01:53.472223 28152 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 [term 2 LEADER]: Committing config change with OpId 2.1096: config changed from index 1094 to 1096, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:53.472856 23726 raft_consensus.cc:2932] T b28e7ed3a23243ee98327f60b2ebdbd4 P 5e1c3e3144384f90b47255623d63575b [term 2 FOLLOWER]: Committing config change with OpId 2.1096: config changed from index 1094 to 1096, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New config: { opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:53.478165 22140 catalog_manager.cc:5484] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 reported cstate change: config changed from index 1094 to 1096, VOTER 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "13c50a9f8b674ac19f095e891401aff9" committed_config { opid_index: 1096 OBSOLETE_local: false peers { permanent_uuid: "5e1c3e3144384f90b47255623d63575b" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 39481 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:53.482982 22523 tablet_service.cc:1514] Processing DeleteTablet for tablet b28e7ed3a23243ee98327f60b2ebdbd4 with delete_type TABLET_DATA_TOMBSTONED (TS 3b1653dc15114e7c873f9cc1a9a3c0a5 not found in new config with opid_index 1096) from {username='slave'} at 127.0.0.1:56510
I20240501 14:01:53.483933 28346 tablet_replica.cc:330] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: stopping tablet replica
I20240501 14:01:53.484136 28346 raft_consensus.cc:2229] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:53.484328 28346 raft_consensus.cc:2258] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:53.486169 28346 ts_tablet_manager.cc:1875] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:53.491686 28346 ts_tablet_manager.cc:1888] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1095
I20240501 14:01:53.491886 28346 log.cc:1196] T b28e7ed3a23243ee98327f60b2ebdbd4 P 3b1653dc15114e7c873f9cc1a9a3c0a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:01:53.492859 22125 catalog_manager.cc:4830] TS 3b1653dc15114e7c873f9cc1a9a3c0a5 (127.16.36.131:38305): tablet b28e7ed3a23243ee98327f60b2ebdbd4 (table rebalance_test_table_4 [id=505c74cf0938498eb9dd7191bfa0e00d]) successfully deleted
I20240501 14:01:53.608094 27460 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:53.634433 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:53.648741 22523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:53.649940 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:53.674799 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:53.676463 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:53.679097 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:53.680198 22392 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:53.819191 27480 consensus_queue.cc:237] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 969, Committed index: 969, Last appended: 2.969, Last appended by leader: 968, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:01:53.820955 23589 raft_consensus.cc:1261] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Refusing update from remote peer 13c50a9f8b674ac19f095e891401aff9: Log matching property violated. Preceding OpId in replica: term: 2 index: 969. Preceding OpId from leader: term: 2 index: 970. (index mismatch)
I20240501 14:01:53.821642 28152 consensus_queue.cc:1035] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 970, Last known committed idx: 969, Time since last communication: 0.000s
I20240501 14:01:53.824119 28151 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 [term 2 LEADER]: Committing config change with OpId 2.970: config changed from index 968 to 970, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:53.824812 23589 raft_consensus.cc:2932] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Committing config change with OpId 2.970: config changed from index 968 to 970, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New config: { opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:53.829360 22139 catalog_manager.cc:5484] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 reported cstate change: config changed from index 968 to 970, VOTER a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130) evicted. New cstate: current_term: 2 leader_uuid: "13c50a9f8b674ac19f095e891401aff9" committed_config { opid_index: 970 OBSOLETE_local: false peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } } }
I20240501 14:01:53.833796 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet c09f45be8c064ecdbdc78d5ce8dbe8bf with delete_type TABLET_DATA_TOMBSTONED (TS a5574cf7d8b54da09d7f589c2c11e97c not found in new config with opid_index 970) from {username='slave'} at 127.0.0.1:41234
I20240501 14:01:53.834553 28405 tablet_replica.cc:330] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: stopping tablet replica
I20240501 14:01:53.834834 28405 raft_consensus.cc:2229] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:53.835081 28405 raft_consensus.cc:2258] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:53.836925 28405 ts_tablet_manager.cc:1875] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:53.842980 28405 ts_tablet_manager.cc:1888] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.969
I20240501 14:01:53.843145 28405 log.cc:1196] T c09f45be8c064ecdbdc78d5ce8dbe8bf P a5574cf7d8b54da09d7f589c2c11e97c: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:01:53.844195 22128 catalog_manager.cc:4830] TS a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): tablet c09f45be8c064ecdbdc78d5ce8dbe8bf (table rebalance_test_table_2 [id=aa4393d7657c42b3b15522ad50965d6c]) successfully deleted
I20240501 14:01:53.968798 27460 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:53.988853 22523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:53.989449 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:54.013855 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:54.019289 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:54.052234 22392 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:54.085574 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:54.118921 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:54.370178 27460 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:54.381302 22523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:54.422076 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:54.438441 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:54.455672 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:54.457233 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:54.458356 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:54.472877 22392 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:54.892216 27460 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:54.908003 22261 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:54.914654 22392 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:01:54.917493 23969 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:54.924438 22523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:54.933053 23838 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:01:54.933571 23705 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:01:54.968041 23572 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e95d660c7bd7435fa08604fea52664ad | 127.16.36.190:35677 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.148:42753 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
13c50a9f8b674ac19f095e891401aff9 | 127.16.36.136:46165 | HEALTHY | <none> | 2 | 0
3b1653dc15114e7c873f9cc1a9a3c0a5 | 127.16.36.131:38305 | HEALTHY | <none> | 3 | 0
474d063159a143fcba0416fa01328bc0 | 127.16.36.129:33963 | HEALTHY | <none> | 4 | 0
5e1c3e3144384f90b47255623d63575b | 127.16.36.133:39481 | HEALTHY | <none> | 2 | 0
90f980f10dfd441289a8ad090e23e200 | 127.16.36.132:45725 | HEALTHY | <none> | 0 | 0
a5574cf7d8b54da09d7f589c2c11e97c | 127.16.36.130:46541 | HEALTHY | <none> | 4 | 0
bcf471e1f6eb42b8a9ae457e6239b89f | 127.16.36.135:41675 | HEALTHY | <none> | 0 | 0
dedba8a189394baca3188f2a5eddbcd4 | 127.16.36.134:35555 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.16.36.129 | experimental | 127.16.36.129:33963
local_ip_for_outbound_sockets | 127.16.36.130 | experimental | 127.16.36.130:46541
local_ip_for_outbound_sockets | 127.16.36.131 | experimental | 127.16.36.131:38305
local_ip_for_outbound_sockets | 127.16.36.132 | experimental | 127.16.36.132:45725
local_ip_for_outbound_sockets | 127.16.36.133 | experimental | 127.16.36.133:39481
local_ip_for_outbound_sockets | 127.16.36.134 | experimental | 127.16.36.134:35555
local_ip_for_outbound_sockets | 127.16.36.135 | experimental | 127.16.36.135:41675
local_ip_for_outbound_sockets | 127.16.36.136 | experimental | 127.16.36.136:46165
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.36.129:33963
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.36.130:46541
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.36.131:38305
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.36.132:45725
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.36.133:39481
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.36.134:35555
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.36.135:41675
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.16.36.136:46165
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.148:42753 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:01:55.046820 16530 log_verifier.cc:126] Checking tablet 161faaa672ce4a0e80495369f1060558
I20240501 14:01:55.391801 16530 log_verifier.cc:177] Verified matching terms for 927 ops in tablet 161faaa672ce4a0e80495369f1060558
I20240501 14:01:55.393748 16530 log_verifier.cc:126] Checking tablet 4ebad65f05b645fdbd0844fee2b23dfe
I20240501 14:01:55.766803 16530 log_verifier.cc:177] Verified matching terms for 926 ops in tablet 4ebad65f05b645fdbd0844fee2b23dfe
I20240501 14:01:55.767904 16530 log_verifier.cc:126] Checking tablet 74e03b7d269c40ff9fbf6457611d849b
I20240501 14:01:56.149725 16530 log_verifier.cc:177] Verified matching terms for 1167 ops in tablet 74e03b7d269c40ff9fbf6457611d849b
I20240501 14:01:56.151347 16530 log_verifier.cc:126] Checking tablet 7bcbcadfd9a34282950350f04b7fcce5
I20240501 14:01:56.554883 16530 log_verifier.cc:177] Verified matching terms for 1112 ops in tablet 7bcbcadfd9a34282950350f04b7fcce5
I20240501 14:01:56.556293 16530 log_verifier.cc:126] Checking tablet 83ae513a62f345d1a6eac152a56936d0
I20240501 14:01:56.885685 16530 log_verifier.cc:177] Verified matching terms for 969 ops in tablet 83ae513a62f345d1a6eac152a56936d0
I20240501 14:01:56.886940 16530 log_verifier.cc:126] Checking tablet 999759caca8344f2b2ae0cff5e204425
I20240501 14:01:57.247431 16530 log_verifier.cc:177] Verified matching terms for 972 ops in tablet 999759caca8344f2b2ae0cff5e204425
I20240501 14:01:57.248592 16530 log_verifier.cc:126] Checking tablet a40334656b4b4333a13956fec5a1053a
I20240501 14:01:57.529688 16530 log_verifier.cc:177] Verified matching terms for 923 ops in tablet a40334656b4b4333a13956fec5a1053a
I20240501 14:01:57.530804 16530 log_verifier.cc:126] Checking tablet b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:01:57.893859 16530 log_verifier.cc:177] Verified matching terms for 1096 ops in tablet b28e7ed3a23243ee98327f60b2ebdbd4
I20240501 14:01:57.896406 16530 log_verifier.cc:126] Checking tablet c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:01:58.240309 16530 log_verifier.cc:177] Verified matching terms for 970 ops in tablet c09f45be8c064ecdbdc78d5ce8dbe8bf
I20240501 14:01:58.241454 16530 log_verifier.cc:126] Checking tablet c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:01:58.581723 16530 log_verifier.cc:177] Verified matching terms for 1097 ops in tablet c40fd7b8bfd04d8ca67a9acf40f74445
I20240501 14:01:58.583106 16530 log_verifier.cc:126] Checking tablet ce58c7fa6c734d3fb453d132540f933c
I20240501 14:01:58.593806 16530 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ce58c7fa6c734d3fb453d132540f933c
I20240501 14:01:58.593973 16530 log_verifier.cc:126] Checking tablet d4716a6313a541b981e5d27b01657956
I20240501 14:01:58.936924 16530 log_verifier.cc:177] Verified matching terms for 982 ops in tablet d4716a6313a541b981e5d27b01657956
I20240501 14:01:58.938014 16530 log_verifier.cc:126] Checking tablet dc19025daec44883afec64b3d12c7c4e
I20240501 14:01:59.230664 16530 log_verifier.cc:177] Verified matching terms for 952 ops in tablet dc19025daec44883afec64b3d12c7c4e
I20240501 14:01:59.231717 16530 log_verifier.cc:126] Checking tablet dd5f6e034ac14cd2864d894fe5b00fd9
I20240501 14:01:59.546123 16530 log_verifier.cc:177] Verified matching terms for 945 ops in tablet dd5f6e034ac14cd2864d894fe5b00fd9
I20240501 14:01:59.547243 16530 log_verifier.cc:126] Checking tablet e6ac874a7f084076bf03ed4167f967e0
I20240501 14:01:59.867484 16530 log_verifier.cc:177] Verified matching terms for 1023 ops in tablet e6ac874a7f084076bf03ed4167f967e0
I20240501 14:01:59.868664 16530 log_verifier.cc:126] Checking tablet fe968af0ff994a9093beca7a614e77b7
I20240501 14:02:00.232636 16530 log_verifier.cc:177] Verified matching terms for 1051 ops in tablet fe968af0ff994a9093beca7a614e77b7
I20240501 14:02:00.237390 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22200
I20240501 14:02:00.254712 22321 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:00.582667 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22200
W20240501 14:02:00.637048 23528 connection.cc:535] server connection from 127.16.36.129:53907 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:00.637876 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22331
I20240501 14:02:00.647948 22453 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:00.976037 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22331
I20240501 14:02:01.023082 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22461
I20240501 14:02:01.029121 22582 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:01.366672 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22461
I20240501 14:02:01.420487 28540 raft_consensus.cc:482] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 474d063159a143fcba0416fa01328bc0)
I20240501 14:02:01.420815 28540 raft_consensus.cc:504] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
W20240501 14:02:01.420985 27414 connection.cc:535] server connection from 127.16.36.131:57007 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:02:01.420524 23923 connection.cc:535] server connection from 127.16.36.131:56177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:02:01.421388 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 23510
I20240501 14:02:01.422453 28540 leader_election.cc:288] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
W20240501 14:02:01.423530 23921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:33963: connect: Connection refused (error 111)
W20240501 14:02:01.424922 23921 leader_election.cc:334] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): Network error: Client connection negotiation failed: client connection to 127.16.36.129:33963: connect: Connection refused (error 111)
I20240501 14:02:01.425098 23921 leader_election.cc:302] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: bcf471e1f6eb42b8a9ae457e6239b89f; no voters: 474d063159a143fcba0416fa01328bc0
I20240501 14:02:01.425527 28540 raft_consensus.cc:2726] T 7bcbcadfd9a34282950350f04b7fcce5 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:01.428135 23631 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:01.523188 28537 raft_consensus.cc:482] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 474d063159a143fcba0416fa01328bc0)
I20240501 14:02:01.523449 28537 raft_consensus.cc:504] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "474d063159a143fcba0416fa01328bc0" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 33963 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:02:01.524248 28537 leader_election.cc:288] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963)
W20240501 14:02:01.525414 23791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.129:33963: connect: Connection refused (error 111)
W20240501 14:02:01.527029 23791 leader_election.cc:334] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 474d063159a143fcba0416fa01328bc0 (127.16.36.129:33963): Network error: Client connection negotiation failed: client connection to 127.16.36.129:33963: connect: Connection refused (error 111)
I20240501 14:02:01.527197 23791 leader_election.cc:302] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: dedba8a189394baca3188f2a5eddbcd4; no voters: 474d063159a143fcba0416fa01328bc0
I20240501 14:02:01.527546 28537 raft_consensus.cc:2726] T d4716a6313a541b981e5d27b01657956 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:01.708091 28548 raft_consensus.cc:482] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5574cf7d8b54da09d7f589c2c11e97c)
I20240501 14:02:01.708310 28548 raft_consensus.cc:504] T 74e03b7d269c40ff9fbf6457611d849b P 90f980f10dfd441289a8ad090e23e200 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "90f980f10dfd441289a8ad090e23e200" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 45725 } }
I20240501 14:02:01.708655 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 23510
I20240501 14:02:01.767128 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 23641
I20240501 14:02:01.773109 23766 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:01.972296 27413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.132:45725: connect: Connection refused (error 111)
W20240501 14:02:01.975315 27413 consensus_peers.cc:485] T c09f45be8c064ecdbdc78d5ce8dbe8bf P 13c50a9f8b674ac19f095e891401aff9 -> Peer 90f980f10dfd441289a8ad090e23e200 (127.16.36.132:45725): Couldn't send request to peer 90f980f10dfd441289a8ad090e23e200. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.132:45725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:02.043321 28553 raft_consensus.cc:482] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a5574cf7d8b54da09d7f589c2c11e97c)
I20240501 14:02:02.043532 28553 raft_consensus.cc:504] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 452 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "bcf471e1f6eb42b8a9ae457e6239b89f" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 41675 } }
I20240501 14:02:02.044272 28553 leader_election.cc:288] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
W20240501 14:02:02.046241 23924 leader_election.cc:334] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): Network error: Client connection negotiation failed: client connection to 127.16.36.130:46541: connect: Connection refused (error 111)
I20240501 14:02:02.046384 23924 leader_election.cc:302] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: bcf471e1f6eb42b8a9ae457e6239b89f; no voters: a5574cf7d8b54da09d7f589c2c11e97c
I20240501 14:02:02.046669 28553 raft_consensus.cc:2726] T 161faaa672ce4a0e80495369f1060558 P bcf471e1f6eb42b8a9ae457e6239b89f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:02.053972 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 23641
I20240501 14:02:02.070844 28537 raft_consensus.cc:482] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a5574cf7d8b54da09d7f589c2c11e97c)
I20240501 14:02:02.071046 28537 raft_consensus.cc:504] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "a5574cf7d8b54da09d7f589c2c11e97c" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 46541 } } peers { permanent_uuid: "dedba8a189394baca3188f2a5eddbcd4" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 35555 } }
I20240501 14:02:02.071700 28537 leader_election.cc:288] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541)
W20240501 14:02:02.074069 23794 leader_election.cc:334] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a5574cf7d8b54da09d7f589c2c11e97c (127.16.36.130:46541): Network error: Client connection negotiation failed: client connection to 127.16.36.130:46541: connect: Connection refused (error 111)
I20240501 14:02:02.074236 23794 leader_election.cc:302] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: dedba8a189394baca3188f2a5eddbcd4; no voters: a5574cf7d8b54da09d7f589c2c11e97c
I20240501 14:02:02.074508 28537 raft_consensus.cc:2726] T fe968af0ff994a9093beca7a614e77b7 P dedba8a189394baca3188f2a5eddbcd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:02:02.108583 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 23776
I20240501 14:02:02.115130 23898 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:02:02.222780 27413 consensus_peers.cc:485] T b28e7ed3a23243ee98327f60b2ebdbd4 P 13c50a9f8b674ac19f095e891401aff9 -> Peer 5e1c3e3144384f90b47255623d63575b (127.16.36.133:39481): Couldn't send request to peer 5e1c3e3144384f90b47255623d63575b. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:39481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:02.407500 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 23776
I20240501 14:02:02.461074 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 23906
I20240501 14:02:02.467085 24030 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:02.740612 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 23906
I20240501 14:02:02.792174 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 27397
I20240501 14:02:02.799065 27521 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:02.994964 28547 raft_consensus.cc:482] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3b1653dc15114e7c873f9cc1a9a3c0a5)
I20240501 14:02:02.995283 28547 raft_consensus.cc:504] T 4ebad65f05b645fdbd0844fee2b23dfe P 13c50a9f8b674ac19f095e891401aff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 926 OBSOLETE_local: false peers { permanent_uuid: "3b1653dc15114e7c873f9cc1a9a3c0a5" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 38305 } } peers { permanent_uuid: "13c50a9f8b674ac19f095e891401aff9" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 46165 } }
I20240501 14:02:02.996320 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 27397
I20240501 14:02:03.027830 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 22110
I20240501 14:02:03.028952 22170 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:03.295723 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 22110
2024-05-01T14:02:03Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (113369 ms)
[----------] 1 test from RebalanceParamTest (113369 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240501 14:02:03.421718 16530 test_util.cc:268] Using random seed: -295977992
I20240501 14:02:03.423588 16530 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:02:03.423664 16530 ts_itest-base.cc:113] --------------
I20240501 14:02:03.423717 16530 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:02:03.423766 16530 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:02:03.423822 16530 ts_itest-base.cc:116] --------------
2024-05-01T14:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:02:03Z Disabled control of system clock
I20240501 14:02:03.488484 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:41971
--webserver_interface=127.16.36.190
--webserver_port=0
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.190:41971
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 14:02:03.872089 28573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:03.872505 28573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:03.872659 28573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:03.882014 28573 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:03.882220 28573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:02:03.882324 28573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:03.882424 28573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:02:03.882495 28573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:02:03.892467 28573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.190:41971
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:41971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:03.894315 28573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:03.897354 28573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:03.911597 28579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:03.912176 28578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:03.912732 28581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:03.913566 28573 server_base.cc:1034] running on GCE node
I20240501 14:02:03.914484 28573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:03.916203 28573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:03.917475 28573 hybrid_clock.cc:648] HybridClock initialized: now 1714572123917415 us; error 57 us; skew 500 ppm
I20240501 14:02:03.917903 28573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:03.920696 28573 webserver.cc:460] Webserver started at http://127.16.36.190:43561/ using document root <none> and password file <none>
I20240501 14:02:03.921583 28573 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:03.921718 28573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:03.922080 28573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:03.925158 28573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1bdce3de4d694d6c98f909a5b3c450ce"
format_stamp: "Formatted at 2024-05-01 14:02:03 on dist-test-slave-9q9t"
I20240501 14:02:03.926106 28573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1bdce3de4d694d6c98f909a5b3c450ce"
format_stamp: "Formatted at 2024-05-01 14:02:03 on dist-test-slave-9q9t"
I20240501 14:02:03.932422 28573 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.006s
I20240501 14:02:03.936708 28587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:03.938249 28573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 14:02:03.938467 28573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1bdce3de4d694d6c98f909a5b3c450ce"
format_stamp: "Formatted at 2024-05-01 14:02:03 on dist-test-slave-9q9t"
I20240501 14:02:03.938795 28573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:03.961999 28573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:03.963089 28573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:03.963402 28573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:03.984052 28573 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.190:41971
I20240501 14:02:03.984143 28638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.190:41971 every 8 connection(s)
I20240501 14:02:03.985972 28573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:02:03.989112 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 28573
I20240501 14:02:03.989370 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:02:03.990829 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:04.004621 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce: Bootstrap starting.
I20240501 14:02:04.008595 28639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:04.009833 28639 log.cc:824] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:04.012681 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce: No bootstrap required, opened a new log
I20240501 14:02:04.019924 28639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 41971 } }
I20240501 14:02:04.020434 28639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:04.020612 28639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdce3de4d694d6c98f909a5b3c450ce, State: Initialized, Role: FOLLOWER
I20240501 14:02:04.021335 28639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 41971 } }
I20240501 14:02:04.021595 28639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:02:04.021740 28639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:02:04.021924 28639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:04.024266 28639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 41971 } }
I20240501 14:02:04.024825 28639 leader_election.cc:302] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bdce3de4d694d6c98f909a5b3c450ce; no voters:
I20240501 14:02:04.025451 28639 leader_election.cc:288] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:02:04.025703 28644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:04.026966 28644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [term 1 LEADER]: Becoming Leader. State: Replica: 1bdce3de4d694d6c98f909a5b3c450ce, State: Running, Role: LEADER
I20240501 14:02:04.027552 28644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 41971 } }
I20240501 14:02:04.027915 28639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:02:04.030879 28645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 41971 } } }
I20240501 14:02:04.030846 28646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdce3de4d694d6c98f909a5b3c450ce. Latest consensus state: current_term: 1 leader_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdce3de4d694d6c98f909a5b3c450ce" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 41971 } } }
I20240501 14:02:04.031323 28645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:04.031319 28646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce [sys.catalog]: This master's current role is: LEADER
I20240501 14:02:04.032409 28653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:02:04.043201 28653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:02:04.056881 28653 catalog_manager.cc:1347] Generated new cluster ID: f6a9e4e3f22c49e18eb21c8277c7e6b5
I20240501 14:02:04.057039 28653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:02:04.075493 28653 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:02:04.076783 28653 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:02:04.088415 28653 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce: Generated new TSK 0
I20240501 14:02:04.089390 28653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:02:04.107586 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.129:0
--local_ip_for_outbound_sockets=127.16.36.129
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:02:04.487322 28663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:04.487674 28663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:04.487828 28663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:04.495674 28663 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:04.495843 28663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:04.496024 28663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.129
I20240501 14:02:04.505975 28663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.129
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:04.507499 28663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:04.509370 28663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:04.523041 28671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:04.523577 28669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:04.524214 28668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:04.524838 28663 server_base.cc:1034] running on GCE node
I20240501 14:02:04.525518 28663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:04.526558 28663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:04.527796 28663 hybrid_clock.cc:648] HybridClock initialized: now 1714572124527734 us; error 56 us; skew 500 ppm
I20240501 14:02:04.528177 28663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:04.530849 28663 webserver.cc:460] Webserver started at http://127.16.36.129:41601/ using document root <none> and password file <none>
I20240501 14:02:04.531633 28663 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:04.531770 28663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:04.532122 28663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:04.534911 28663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "343a85ba3b514663af5320cd14a9e144"
format_stamp: "Formatted at 2024-05-01 14:02:04 on dist-test-slave-9q9t"
I20240501 14:02:04.535707 28663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "343a85ba3b514663af5320cd14a9e144"
format_stamp: "Formatted at 2024-05-01 14:02:04 on dist-test-slave-9q9t"
I20240501 14:02:04.541332 28663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:04.545100 28677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:04.546618 28663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:04.546823 28663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "343a85ba3b514663af5320cd14a9e144"
format_stamp: "Formatted at 2024-05-01 14:02:04 on dist-test-slave-9q9t"
I20240501 14:02:04.547119 28663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:04.569315 28663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:04.570271 28663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:04.570549 28663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:04.571677 28663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:04.573333 28663 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:04.573429 28663 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:04.573521 28663 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:04.573580 28663 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:04.614456 28663 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.129:41051
I20240501 14:02:04.614503 28789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.129:41051 every 8 connection(s)
I20240501 14:02:04.615947 28663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:02:04.619213 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 28663
I20240501 14:02:04.619447 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:02:04.623440 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.130:0
--local_ip_for_outbound_sockets=127.16.36.130
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:04.630898 28790 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:04.631234 28790 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:04.632050 28790 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:04.634521 28604 ts_manager.cc:194] Registered new tserver with Master: 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:04.636875 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.129:59461
W20240501 14:02:05.006487 28794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:05.006944 28794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:05.007107 28794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:05.015537 28794 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:05.015681 28794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:05.015805 28794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.130
I20240501 14:02:05.025532 28794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.130
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:05.026932 28794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:05.028812 28794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:05.042743 28802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:05.043983 28800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:05.043951 28799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:05.045670 28794 server_base.cc:1034] running on GCE node
I20240501 14:02:05.046365 28794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:05.047473 28794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:05.048687 28794 hybrid_clock.cc:648] HybridClock initialized: now 1714572125048644 us; error 40 us; skew 500 ppm
I20240501 14:02:05.049053 28794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:05.051683 28794 webserver.cc:460] Webserver started at http://127.16.36.130:43207/ using document root <none> and password file <none>
I20240501 14:02:05.052528 28794 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:05.052644 28794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:05.053047 28794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:05.056161 28794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cd3c04ebb6c84b15a1b3adc748737094"
format_stamp: "Formatted at 2024-05-01 14:02:05 on dist-test-slave-9q9t"
I20240501 14:02:05.057075 28794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cd3c04ebb6c84b15a1b3adc748737094"
format_stamp: "Formatted at 2024-05-01 14:02:05 on dist-test-slave-9q9t"
I20240501 14:02:05.063037 28794 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240501 14:02:05.067157 28808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:05.068842 28794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:02:05.069065 28794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cd3c04ebb6c84b15a1b3adc748737094"
format_stamp: "Formatted at 2024-05-01 14:02:05 on dist-test-slave-9q9t"
I20240501 14:02:05.069307 28794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:05.111589 28794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:05.112565 28794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:05.112866 28794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:05.113972 28794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:05.115772 28794 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:05.115906 28794 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:05.116036 28794 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:05.116112 28794 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:05.148847 28794 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.130:37325
I20240501 14:02:05.148950 28920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.130:37325 every 8 connection(s)
I20240501 14:02:05.150502 28794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:02:05.157021 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 28794
I20240501 14:02:05.157253 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:02:05.160970 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.131:0
--local_ip_for_outbound_sockets=127.16.36.131
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:05.165930 28921 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:05.166278 28921 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:05.167155 28921 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:05.168869 28604 ts_manager.cc:194] Registered new tserver with Master: cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:05.170038 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.130:46699
W20240501 14:02:05.550817 28925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:05.551139 28925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:05.551295 28925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:05.559549 28925 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:05.559712 28925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:05.559903 28925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.131
I20240501 14:02:05.569736 28925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.131
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:05.571269 28925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:05.573143 28925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:05.587200 28930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:05.587815 28931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:05.588276 28933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:05.589176 28925 server_base.cc:1034] running on GCE node
I20240501 14:02:05.589962 28925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:05.591122 28925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:05.592388 28925 hybrid_clock.cc:648] HybridClock initialized: now 1714572125592336 us; error 43 us; skew 500 ppm
I20240501 14:02:05.592713 28925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:05.595220 28925 webserver.cc:460] Webserver started at http://127.16.36.131:45241/ using document root <none> and password file <none>
I20240501 14:02:05.595985 28925 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:05.596102 28925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:05.596403 28925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:05.599226 28925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7e7bc120bd74479ab378f779778fd1e8"
format_stamp: "Formatted at 2024-05-01 14:02:05 on dist-test-slave-9q9t"
I20240501 14:02:05.600100 28925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7e7bc120bd74479ab378f779778fd1e8"
format_stamp: "Formatted at 2024-05-01 14:02:05 on dist-test-slave-9q9t"
I20240501 14:02:05.606115 28925 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240501 14:02:05.611513 28939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:05.613132 28925 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:02:05.613353 28925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7e7bc120bd74479ab378f779778fd1e8"
format_stamp: "Formatted at 2024-05-01 14:02:05 on dist-test-slave-9q9t"
I20240501 14:02:05.613590 28925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:05.640400 28790 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
I20240501 14:02:05.664135 28925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:05.665103 28925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:05.665408 28925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:05.666447 28925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:05.668061 28925 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:05.668164 28925 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:05.668290 28925 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:05.668375 28925 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:05.701951 28925 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.131:41159
I20240501 14:02:05.702031 29051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.131:41159 every 8 connection(s)
I20240501 14:02:05.703511 28925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:02:05.713812 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 28925
I20240501 14:02:05.714031 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:02:05.717628 29052 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:05.718006 29052 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:05.717985 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:0
--local_ip_for_outbound_sockets=127.16.36.132
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:05.718952 29052 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:05.720659 28604 ts_manager.cc:194] Registered new tserver with Master: 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:05.721638 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.131:37423
W20240501 14:02:06.091691 29056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:06.092056 29056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:06.092190 29056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:06.099957 29056 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:06.100142 29056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:06.100328 29056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 14:02:06.110417 29056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:06.111972 29056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:06.113826 29056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:06.127103 29061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:06.127863 29062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:06.128502 29064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:06.129230 29056 server_base.cc:1034] running on GCE node
I20240501 14:02:06.129886 29056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:06.130903 29056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:06.132109 29056 hybrid_clock.cc:648] HybridClock initialized: now 1714572126132045 us; error 61 us; skew 500 ppm
I20240501 14:02:06.132419 29056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:06.134843 29056 webserver.cc:460] Webserver started at http://127.16.36.132:46669/ using document root <none> and password file <none>
I20240501 14:02:06.135622 29056 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:06.135725 29056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:06.136011 29056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:06.138980 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3a3972b170c54344a4ba61f176f837ba"
format_stamp: "Formatted at 2024-05-01 14:02:06 on dist-test-slave-9q9t"
I20240501 14:02:06.139801 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3a3972b170c54344a4ba61f176f837ba"
format_stamp: "Formatted at 2024-05-01 14:02:06 on dist-test-slave-9q9t"
I20240501 14:02:06.145540 29056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:06.149494 29070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:06.151072 29056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:02:06.151257 29056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3a3972b170c54344a4ba61f176f837ba"
format_stamp: "Formatted at 2024-05-01 14:02:06 on dist-test-slave-9q9t"
I20240501 14:02:06.151460 29056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:06.168632 29056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:06.169591 29056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:06.169879 29056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:06.170980 29056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:06.172684 29056 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:06.172780 29056 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:06.172901 29056 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:06.172969 29056 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:06.173128 28921 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
I20240501 14:02:06.207675 29056 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:42827
I20240501 14:02:06.207729 29182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:42827 every 8 connection(s)
I20240501 14:02:06.209115 29056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:02:06.210279 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 29056
I20240501 14:02:06.210489 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:02:06.214555 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:0
--local_ip_for_outbound_sockets=127.16.36.133
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:06.231989 29183 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:06.232697 29183 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:06.233711 29183 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:06.235452 28604 ts_manager.cc:194] Registered new tserver with Master: 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:06.236305 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:44935
W20240501 14:02:06.584348 29186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:06.584683 29186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:06.584841 29186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:06.593057 29186 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:06.593192 29186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:06.593366 29186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 14:02:06.602933 29186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:06.604337 29186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:06.606153 29186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:06.620836 29192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:06.621452 29194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:06.622092 29191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:06.624794 29186 server_base.cc:1034] running on GCE node
I20240501 14:02:06.625527 29186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:06.626569 29186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:06.627810 29186 hybrid_clock.cc:648] HybridClock initialized: now 1714572126627742 us; error 64 us; skew 500 ppm
I20240501 14:02:06.628177 29186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:06.630721 29186 webserver.cc:460] Webserver started at http://127.16.36.133:34471/ using document root <none> and password file <none>
I20240501 14:02:06.631525 29186 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:06.631647 29186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:06.631968 29186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:06.634730 29186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "26acba62745c48e3ba8180748d55b211"
format_stamp: "Formatted at 2024-05-01 14:02:06 on dist-test-slave-9q9t"
I20240501 14:02:06.635572 29186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "26acba62745c48e3ba8180748d55b211"
format_stamp: "Formatted at 2024-05-01 14:02:06 on dist-test-slave-9q9t"
I20240501 14:02:06.641172 29186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:02:06.644927 29200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:06.646499 29186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:06.646716 29186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "26acba62745c48e3ba8180748d55b211"
format_stamp: "Formatted at 2024-05-01 14:02:06 on dist-test-slave-9q9t"
I20240501 14:02:06.646939 29186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:06.667349 29186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:06.668272 29186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:06.668563 29186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:06.669605 29186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:06.671244 29186 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:06.671346 29186 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:06.671478 29186 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:06.671552 29186 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:06.705117 29186 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:40601
I20240501 14:02:06.705180 29312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:40601 every 8 connection(s)
I20240501 14:02:06.706887 29186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:02:06.708427 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 29186
I20240501 14:02:06.708587 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:02:06.712309 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.134:0
--local_ip_for_outbound_sockets=127.16.36.134
--webserver_interface=127.16.36.134
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:06.721037 29313 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:06.721396 29313 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:06.722354 29313 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:06.723953 28604 ts_manager.cc:194] Registered new tserver with Master: 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:02:06.725112 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:49051
I20240501 14:02:06.736729 29052 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
W20240501 14:02:07.082546 29316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:07.082944 29316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:07.083088 29316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:07.090816 29316 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:07.090974 29316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:07.091135 29316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.134
I20240501 14:02:07.100482 29316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.134
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.134
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:07.101924 29316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:07.103964 29316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:07.116891 29321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:07.117558 29322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:07.118212 29324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:07.119011 29316 server_base.cc:1034] running on GCE node
I20240501 14:02:07.119844 29316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:07.120909 29316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:07.122176 29316 hybrid_clock.cc:648] HybridClock initialized: now 1714572127122129 us; error 42 us; skew 500 ppm
I20240501 14:02:07.122540 29316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:07.125033 29316 webserver.cc:460] Webserver started at http://127.16.36.134:34609/ using document root <none> and password file <none>
I20240501 14:02:07.125815 29316 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:07.125918 29316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:07.126212 29316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:07.129074 29316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "8fc5d8222c164a679e7a99966da1eaaf"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:07.129881 29316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "8fc5d8222c164a679e7a99966da1eaaf"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:07.135491 29316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 14:02:07.139183 29330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:07.140700 29316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 14:02:07.140888 29316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8fc5d8222c164a679e7a99966da1eaaf"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:07.141114 29316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:07.165906 29316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:07.166934 29316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:07.167229 29316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:07.168258 29316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:07.169836 29316 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:07.169939 29316 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:07.170064 29316 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:07.170140 29316 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:07.203801 29316 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.134:44241
I20240501 14:02:07.203850 29442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.134:44241 every 8 connection(s)
I20240501 14:02:07.205245 29316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:02:07.213544 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 29316
I20240501 14:02:07.213749 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:02:07.217484 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.135:0
--local_ip_for_outbound_sockets=127.16.36.135
--webserver_interface=127.16.36.135
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:07.220142 29443 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:07.220543 29443 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:07.221674 29443 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:07.229089 28604 ts_manager.cc:194] Registered new tserver with Master: 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:02:07.230350 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.134:55697
I20240501 14:02:07.239159 29183 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
W20240501 14:02:07.585350 29446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:07.585692 29446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:07.585824 29446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:07.593683 29446 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:07.593827 29446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:07.593966 29446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.135
I20240501 14:02:07.603547 29446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.135
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.135
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:07.604959 29446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:07.606820 29446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:07.621201 29451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:07.621795 29452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:07.622707 29454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:07.623253 29446 server_base.cc:1034] running on GCE node
I20240501 14:02:07.623955 29446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:07.625066 29446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:07.626289 29446 hybrid_clock.cc:648] HybridClock initialized: now 1714572127626230 us; error 50 us; skew 500 ppm
I20240501 14:02:07.626684 29446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:07.629630 29446 webserver.cc:460] Webserver started at http://127.16.36.135:44841/ using document root <none> and password file <none>
I20240501 14:02:07.630468 29446 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:07.630584 29446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:07.630975 29446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:07.633846 29446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a0e901103604414c829b40edf9f24c28"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:07.634711 29446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a0e901103604414c829b40edf9f24c28"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:07.640534 29446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240501 14:02:07.644666 29460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:07.646471 29446 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:07.646703 29446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a0e901103604414c829b40edf9f24c28"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:07.646975 29446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:07.663452 29446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:07.664301 29446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:07.664552 29446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:07.665525 29446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:07.667066 29446 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:07.667160 29446 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:07.667244 29446 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:07.667302 29446 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:07.698658 29446 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.135:42127
I20240501 14:02:07.698736 29572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.135:42127 every 8 connection(s)
I20240501 14:02:07.700160 29446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:02:07.710462 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 29446
I20240501 14:02:07.710702 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:02:07.714573 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.136:0
--local_ip_for_outbound_sockets=127.16.36.136
--webserver_interface=127.16.36.136
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:07.715492 29573 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:07.715824 29573 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:07.716625 29573 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:07.718093 28604 ts_manager.cc:194] Registered new tserver with Master: a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:02:07.719033 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.135:59747
I20240501 14:02:07.728284 29313 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
W20240501 14:02:08.085057 29576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:08.085390 29576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:08.085494 29576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:08.093117 29576 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:08.093263 29576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:08.093384 29576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.136
I20240501 14:02:08.102532 29576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.136
--webserver_port=0
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.136
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:08.103879 29576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:08.105638 29576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:08.118805 29581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:08.119395 29582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:08.119948 29584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:08.120718 29576 server_base.cc:1034] running on GCE node
I20240501 14:02:08.121369 29576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:08.122443 29576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:08.123648 29576 hybrid_clock.cc:648] HybridClock initialized: now 1714572128123567 us; error 102 us; skew 500 ppm
I20240501 14:02:08.123998 29576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:08.126775 29576 webserver.cc:460] Webserver started at http://127.16.36.136:43511/ using document root <none> and password file <none>
I20240501 14:02:08.127650 29576 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:08.127755 29576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:08.128111 29576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:02:08.131013 29576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "826ccf8358b6453d88bcb132290b82a5"
format_stamp: "Formatted at 2024-05-01 14:02:08 on dist-test-slave-9q9t"
I20240501 14:02:08.131804 29576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "826ccf8358b6453d88bcb132290b82a5"
format_stamp: "Formatted at 2024-05-01 14:02:08 on dist-test-slave-9q9t"
I20240501 14:02:08.137641 29576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:02:08.141539 29590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:08.143168 29576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:08.143347 29576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "826ccf8358b6453d88bcb132290b82a5"
format_stamp: "Formatted at 2024-05-01 14:02:08 on dist-test-slave-9q9t"
I20240501 14:02:08.143568 29576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:08.165426 29576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:08.166332 29576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:08.166821 29576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:08.167907 29576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:08.169517 29576 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:08.169608 29576 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:08.169713 29576 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:08.169780 29576 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:08.200881 29576 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.136:36399
I20240501 14:02:08.200932 29702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.136:36399 every 8 connection(s)
I20240501 14:02:08.202311 29576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:02:08.208066 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 29576
I20240501 14:02:08.208278 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:02:08.225080 29703 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:08.225498 29703 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:08.226655 29703 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:08.228427 28604 ts_manager.cc:194] Registered new tserver with Master: 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
I20240501 14:02:08.229764 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.136:35497
I20240501 14:02:08.234023 29443 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
I20240501 14:02:08.238231 16530 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:02:08.262341 16530 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:02:08.262511 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 29186
I20240501 14:02:08.276252 29308 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:08.360613 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 29186
I20240501 14:02:08.378926 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 29316
I20240501 14:02:08.388669 29438 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:08.474965 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 29316
I20240501 14:02:08.493343 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 29446
I20240501 14:02:08.502125 29568 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:08.585778 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 29446
I20240501 14:02:08.602836 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 29576
I20240501 14:02:08.612080 29698 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:02:08.700667 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 29576
I20240501 14:02:12.472111 28604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:12.523244 28723 tablet_service.cc:1467] Processing CreateTablet for tablet 86bfedcdabe44e7ab135674163935786 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.523963 28725 tablet_service.cc:1467] Processing CreateTablet for tablet 60b3db79ff754e8bbe9e5865f7104312 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.524036 28856 tablet_service.cc:1467] Processing CreateTablet for tablet 60b3db79ff754e8bbe9e5865f7104312 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.525007 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bfedcdabe44e7ab135674163935786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.524870 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 86bfedcdabe44e7ab135674163935786 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.525084 29118 tablet_service.cc:1467] Processing CreateTablet for tablet 60b3db79ff754e8bbe9e5865f7104312 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.525885 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b3db79ff754e8bbe9e5865f7104312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.526115 28855 tablet_service.cc:1467] Processing CreateTablet for tablet bb712628c7764a138f5425d14b724d9e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.526490 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bfedcdabe44e7ab135674163935786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.526835 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb712628c7764a138f5425d14b724d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.527046 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b3db79ff754e8bbe9e5865f7104312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.523244 28724 tablet_service.cc:1467] Processing CreateTablet for tablet bb712628c7764a138f5425d14b724d9e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.528908 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb712628c7764a138f5425d14b724d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.532322 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b3db79ff754e8bbe9e5865f7104312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.549407 28986 tablet_service.cc:1467] Processing CreateTablet for tablet 86bfedcdabe44e7ab135674163935786 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.551476 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bfedcdabe44e7ab135674163935786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.548697 28987 tablet_service.cc:1467] Processing CreateTablet for tablet bb712628c7764a138f5425d14b724d9e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:12.553370 28987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb712628c7764a138f5425d14b724d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:12.553328 29732 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:12.557796 29732 tablet_bootstrap.cc:654] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.559396 29732 log.cc:824] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:12.579289 29735 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:12.586246 29735 tablet_bootstrap.cc:654] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.588248 29733 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:12.589682 29736 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:12.591106 29735 log.cc:824] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:12.592079 29732 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:12.592453 29732 ts_tablet_manager.cc:1367] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.024s
I20240501 14:02:12.593782 29736 tablet_bootstrap.cc:654] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.593840 29733 tablet_bootstrap.cc:654] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.595274 29736 log.cc:824] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:12.595455 29733 log.cc:824] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:12.600853 29733 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:12.600853 29735 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:12.601202 29733 ts_tablet_manager.cc:1367] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 14:02:12.601194 29735 ts_tablet_manager.cc:1367] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.004s
I20240501 14:02:12.602542 29732 raft_consensus.cc:348] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.603144 29732 raft_consensus.cc:374] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.603273 29732 raft_consensus.cc:729] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.603861 29736 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:12.604249 29736 ts_tablet_manager.cc:1367] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240501 14:02:12.604095 29732 consensus_queue.cc:260] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.611523 29735 raft_consensus.cc:348] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.612074 29735 raft_consensus.cc:374] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.612190 29735 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.611979 29733 raft_consensus.cc:348] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.612498 29733 raft_consensus.cc:374] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.612612 29733 raft_consensus.cc:729] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.613013 29735 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.613394 29733 consensus_queue.cc:260] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.616124 29732 ts_tablet_manager.cc:1398] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
I20240501 14:02:12.617121 29736 raft_consensus.cc:348] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.617741 29736 raft_consensus.cc:374] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.617905 29736 raft_consensus.cc:729] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.618669 29732 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:12.618868 29736 consensus_queue.cc:260] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.621203 29732 tablet_bootstrap.cc:654] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
W20240501 14:02:12.624492 28791 tablet.cc:2369] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:12.627247 29733 ts_tablet_manager.cc:1398] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.026s user 0.016s sys 0.000s
I20240501 14:02:12.628029 29733 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:12.630824 29733 tablet_bootstrap.cc:654] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.636366 29735 ts_tablet_manager.cc:1398] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.035s user 0.010s sys 0.018s
I20240501 14:02:12.636862 29735 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:12.639065 29735 tablet_bootstrap.cc:654] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.647702 29735 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:12.647873 29735 ts_tablet_manager.cc:1367] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240501 14:02:12.648881 29735 raft_consensus.cc:348] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.649134 29735 raft_consensus.cc:374] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.649245 29735 raft_consensus.cc:729] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.649566 29735 consensus_queue.cc:260] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.650254 29735 ts_tablet_manager.cc:1398] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:02:12.654764 29732 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:12.654969 29732 ts_tablet_manager.cc:1367] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.003s
I20240501 14:02:12.656276 29733 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:12.656453 29733 ts_tablet_manager.cc:1367] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.008s
I20240501 14:02:12.656390 29732 raft_consensus.cc:348] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.656701 29732 raft_consensus.cc:374] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.656805 29732 raft_consensus.cc:729] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.656895 29736 ts_tablet_manager.cc:1398] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.052s user 0.025s sys 0.023s
I20240501 14:02:12.657092 29732 consensus_queue.cc:260] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.657349 29736 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:12.657891 29732 ts_tablet_manager.cc:1398] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:12.658237 29732 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:12.658838 29733 raft_consensus.cc:348] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
W20240501 14:02:12.659005 28922 tablet.cc:2369] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:12.659099 29733 raft_consensus.cc:374] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.659144 29736 tablet_bootstrap.cc:654] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.659219 29733 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.659476 29733 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.660260 29732 tablet_bootstrap.cc:654] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:12.662189 29733 ts_tablet_manager.cc:1398] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:12.667390 29732 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:12.667614 29732 ts_tablet_manager.cc:1367] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 14:02:12.667894 29736 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:12.668049 29736 ts_tablet_manager.cc:1367] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 14:02:12.668519 29732 raft_consensus.cc:348] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.668785 29732 raft_consensus.cc:374] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.668866 29732 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.669157 29736 raft_consensus.cc:348] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.669168 29732 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.669392 29736 raft_consensus.cc:374] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:12.669484 29736 raft_consensus.cc:729] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:12.669818 29732 ts_tablet_manager.cc:1398] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:12.669725 29736 consensus_queue.cc:260] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.670331 29736 ts_tablet_manager.cc:1398] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:02:12.671881 29742 raft_consensus.cc:482] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:12.672145 29742 raft_consensus.cc:504] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.674468 29742 leader_election.cc:288] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:12.682343 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb712628c7764a138f5425d14b724d9e" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:12.682931 28745 raft_consensus.cc:2454] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:12.683629 28811 leader_election.cc:302] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:12.684123 29742 raft_consensus.cc:2781] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:12.684288 29742 raft_consensus.cc:482] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:12.684254 29741 raft_consensus.cc:482] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:12.684409 29742 raft_consensus.cc:3037] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.684521 29741 raft_consensus.cc:504] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.686308 29741 leader_election.cc:288] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:12.686728 29742 raft_consensus.cc:504] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.687522 29742 leader_election.cc:288] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:12.687330 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb712628c7764a138f5425d14b724d9e" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:12.688009 29007 raft_consensus.cc:2454] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:12.688376 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb712628c7764a138f5425d14b724d9e" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:12.688616 28745 raft_consensus.cc:3037] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.689723 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb712628c7764a138f5425d14b724d9e" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:12.689935 29007 raft_consensus.cc:3037] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.692062 28745 raft_consensus.cc:2454] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:12.692574 28811 leader_election.cc:302] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:12.692960 29742 raft_consensus.cc:2781] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:12.693596 29007 raft_consensus.cc:2454] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:12.694947 29742 raft_consensus.cc:686] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:02:12.695544 29742 consensus_queue.cc:237] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:12.698441 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:12.698745 28745 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:12.698729 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:12.699149 28876 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:12.699717 29071 leader_election.cc:302] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:12.700266 29741 raft_consensus.cc:2781] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:12.700428 29741 raft_consensus.cc:482] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:12.700531 29741 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.701900 28602 catalog_manager.cc:5484] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:12.702925 29741 raft_consensus.cc:504] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.703663 29741 leader_election.cc:288] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:12.704128 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:12.704349 28876 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.704547 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:12.704845 28745 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.706336 29741 raft_consensus.cc:482] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:12.706521 29741 raft_consensus.cc:504] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.707736 28876 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:12.708019 28745 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:12.708249 29071 leader_election.cc:302] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:12.709571 29741 leader_election.cc:288] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:12.709800 29741 raft_consensus.cc:2781] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:12.710045 29741 raft_consensus.cc:686] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
W20240501 14:02:12.710811 29053 tablet.cc:2369] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:12.711848 29741 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.714565 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bfedcdabe44e7ab135674163935786" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:12.714876 28745 raft_consensus.cc:2454] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:12.715369 29073 leader_election.cc:302] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
W20240501 14:02:12.716286 29184 tablet.cc:2369] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:02:12.719158 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bfedcdabe44e7ab135674163935786" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:12.719532 29007 raft_consensus.cc:2454] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:12.722293 29757 raft_consensus.cc:2781] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:12.722433 29757 raft_consensus.cc:482] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:12.722577 29757 raft_consensus.cc:3037] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.724889 29757 raft_consensus.cc:504] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.726135 29757 leader_election.cc:288] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:12.726152 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bfedcdabe44e7ab135674163935786" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:12.726258 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bfedcdabe44e7ab135674163935786" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:12.726475 28745 raft_consensus.cc:3037] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.726469 29007 raft_consensus.cc:3037] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:12.727639 28602 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:02:12.729435 29007 raft_consensus.cc:2454] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:12.729463 28745 raft_consensus.cc:2454] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:12.729970 29072 leader_election.cc:302] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:12.730665 29757 raft_consensus.cc:2781] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:12.730880 29757 raft_consensus.cc:686] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:12.731305 29757 consensus_queue.cc:237] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:12.733968 28602 catalog_manager.cc:5484] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
W20240501 14:02:13.138756 29759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:13.139110 29759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:13.146997 29759 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:02:13.224745 29757 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:13.238611 29757 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:13.265034 29757 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:13.271478 29787 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:13.289770 29784 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:13.308424 29795 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:13.414520 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 56245.5us
INSERT report
rows total: 2000
time total: 304.309 ms
time per row: 0.152155 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:02:14.545379 28596 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:14.555191 28855 tablet_service.cc:1467] Processing CreateTablet for tablet 2020fe37059a4e74bb45190398e4e177 (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.555281 29118 tablet_service.cc:1467] Processing CreateTablet for tablet 2020fe37059a4e74bb45190398e4e177 (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.555712 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2020fe37059a4e74bb45190398e4e177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.555712 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2020fe37059a4e74bb45190398e4e177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.555992 28987 tablet_service.cc:1467] Processing CreateTablet for tablet 2020fe37059a4e74bb45190398e4e177 (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.556430 28987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2020fe37059a4e74bb45190398e4e177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.556553 28985 tablet_service.cc:1467] Processing CreateTablet for tablet fb04ec77da194832aa63d69aba307de3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.556922 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb04ec77da194832aa63d69aba307de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.562750 28986 tablet_service.cc:1467] Processing CreateTablet for tablet f516f01e242d4b8789130201f748f3ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.563645 28856 tablet_service.cc:1467] Processing CreateTablet for tablet f516f01e242d4b8789130201f748f3ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.563925 29117 tablet_service.cc:1467] Processing CreateTablet for tablet f516f01e242d4b8789130201f748f3ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.564110 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f516f01e242d4b8789130201f748f3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.564373 28854 tablet_service.cc:1467] Processing CreateTablet for tablet fb04ec77da194832aa63d69aba307de3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.564462 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f516f01e242d4b8789130201f748f3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.564742 28854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb04ec77da194832aa63d69aba307de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.571626 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f516f01e242d4b8789130201f748f3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.572021 28723 tablet_service.cc:1467] Processing CreateTablet for tablet fb04ec77da194832aa63d69aba307de3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:14.572513 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb04ec77da194832aa63d69aba307de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:14.576697 29832 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:14.578506 29832 tablet_bootstrap.cc:654] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.580742 29834 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:14.580747 29833 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:14.582536 29834 tablet_bootstrap.cc:654] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.582533 29833 tablet_bootstrap.cc:654] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.584517 29832 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:14.584801 29832 ts_tablet_manager.cc:1367] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:02:14.585958 29832 raft_consensus.cc:348] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.586427 29832 raft_consensus.cc:374] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.586597 29832 raft_consensus.cc:729] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.587018 29832 consensus_queue.cc:260] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.587738 29833 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:14.587792 29834 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:14.587924 29834 ts_tablet_manager.cc:1367] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:02:14.587913 29833 ts_tablet_manager.cc:1367] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240501 14:02:14.588625 29832 ts_tablet_manager.cc:1398] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:14.588706 29834 raft_consensus.cc:348] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.588706 29833 raft_consensus.cc:348] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.588917 29832 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:14.588915 29833 raft_consensus.cc:374] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.588912 29834 raft_consensus.cc:374] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.589002 29834 raft_consensus.cc:729] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.589002 29833 raft_consensus.cc:729] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.589257 29833 consensus_queue.cc:260] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.589257 29834 consensus_queue.cc:260] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.589818 29833 ts_tablet_manager.cc:1398] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:14.590006 29836 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:14.590099 29833 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:14.590654 29832 tablet_bootstrap.cc:654] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.591756 29836 tablet_bootstrap.cc:654] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.592267 29833 tablet_bootstrap.cc:654] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.600965 29834 ts_tablet_manager.cc:1398] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.013s user 0.000s sys 0.006s
I20240501 14:02:14.601406 29834 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:14.603754 29834 tablet_bootstrap.cc:654] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.610944 29833 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:14.611320 29833 ts_tablet_manager.cc:1367] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.012s
I20240501 14:02:14.612486 29833 raft_consensus.cc:348] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.612903 29833 raft_consensus.cc:374] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.613055 29833 raft_consensus.cc:729] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.613102 29836 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:14.613315 29836 ts_tablet_manager.cc:1367] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.006s
I20240501 14:02:14.613580 29833 consensus_queue.cc:260] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.614599 29833 ts_tablet_manager.cc:1398] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:14.625978 29836 raft_consensus.cc:348] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.626363 29836 raft_consensus.cc:374] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.626513 29836 raft_consensus.cc:729] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.626874 29836 consensus_queue.cc:260] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.627091 29834 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:14.627418 29834 ts_tablet_manager.cc:1367] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.013s
I20240501 14:02:14.628357 29834 raft_consensus.cc:348] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.628659 29834 raft_consensus.cc:374] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.628712 29832 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:14.628787 29834 raft_consensus.cc:729] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.629175 29834 consensus_queue.cc:260] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.629540 29832 ts_tablet_manager.cc:1367] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.013s
I20240501 14:02:14.629820 29834 ts_tablet_manager.cc:1398] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:14.630424 29836 ts_tablet_manager.cc:1398] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20240501 14:02:14.630550 29834 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:14.630501 29832 raft_consensus.cc:348] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.630853 29832 raft_consensus.cc:374] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.630977 29832 raft_consensus.cc:729] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.631304 29832 consensus_queue.cc:260] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.632994 29832 ts_tablet_manager.cc:1398] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:14.633373 29832 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:14.633998 29834 tablet_bootstrap.cc:654] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.635280 29832 tablet_bootstrap.cc:654] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:14.639420 29811 raft_consensus.cc:482] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:14.639619 29811 raft_consensus.cc:504] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.640095 29832 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:14.640244 29832 ts_tablet_manager.cc:1367] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:02:14.640362 29834 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:14.640408 29811 leader_election.cc:288] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:14.640601 29834 ts_tablet_manager.cc:1367] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240501 14:02:14.641047 29832 raft_consensus.cc:348] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.641264 29832 raft_consensus.cc:374] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.641366 29832 raft_consensus.cc:729] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.641567 29834 raft_consensus.cc:348] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.641649 29832 consensus_queue.cc:260] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.641835 29834 raft_consensus.cc:374] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:14.641952 29834 raft_consensus.cc:729] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:14.642031 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:14.642256 29832 ts_tablet_manager.cc:1398] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:14.642256 29834 consensus_queue.cc:260] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.642277 29007 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:14.642796 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:14.642905 29834 ts_tablet_manager.cc:1398] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:14.643021 28876 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:14.643440 29072 leader_election.cc:302] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:14.643754 29811 raft_consensus.cc:2781] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:14.643901 29811 raft_consensus.cc:482] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:14.643997 29811 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.646550 29811 raft_consensus.cc:504] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.647218 29811 leader_election.cc:288] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:14.647671 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:14.647708 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:14.647869 29007 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.647893 28876 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.650034 28876 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:14.650048 29007 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:14.650516 29071 leader_election.cc:302] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:14.650894 29811 raft_consensus.cc:2781] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:14.651268 29811 raft_consensus.cc:686] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:14.651618 29811 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:14.654011 28598 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:14.731003 29839 raft_consensus.cc:482] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:14.731245 29839 raft_consensus.cc:504] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.732894 29839 leader_election.cc:288] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:14.753875 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb04ec77da194832aa63d69aba307de3" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:14.754194 28876 raft_consensus.cc:2454] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:14.754153 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb04ec77da194832aa63d69aba307de3" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:14.754415 28743 raft_consensus.cc:2454] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:14.754779 28940 leader_election.cc:302] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:14.755211 29839 raft_consensus.cc:2781] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:14.755347 29839 raft_consensus.cc:482] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:14.755453 29839 raft_consensus.cc:3037] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.757556 29839 raft_consensus.cc:504] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.758153 29839 leader_election.cc:288] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:14.758644 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb04ec77da194832aa63d69aba307de3" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:14.758708 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb04ec77da194832aa63d69aba307de3" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:14.758857 28876 raft_consensus.cc:3037] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.758911 28743 raft_consensus.cc:3037] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.760731 28876 raft_consensus.cc:2454] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:14.760885 28743 raft_consensus.cc:2454] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:14.761168 28940 leader_election.cc:302] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:14.761485 29839 raft_consensus.cc:2781] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:14.761911 29839 raft_consensus.cc:686] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:14.762352 29839 consensus_queue.cc:237] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:14.765713 28598 catalog_manager.cc:5484] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:02:14.924072 29811 raft_consensus.cc:482] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:14.924310 29811 raft_consensus.cc:504] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.924975 29811 leader_election.cc:288] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:14.925640 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f516f01e242d4b8789130201f748f3ea" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:14.925629 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f516f01e242d4b8789130201f748f3ea" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:14.925885 29007 raft_consensus.cc:2454] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:14.925889 28876 raft_consensus.cc:2454] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:14.926374 29072 leader_election.cc:302] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:14.926733 29811 raft_consensus.cc:2781] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:14.926842 29811 raft_consensus.cc:482] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:14.926966 29811 raft_consensus.cc:3037] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.929064 29811 raft_consensus.cc:504] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.929742 29811 leader_election.cc:288] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:14.930326 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f516f01e242d4b8789130201f748f3ea" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:14.930327 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f516f01e242d4b8789130201f748f3ea" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:14.930604 28876 raft_consensus.cc:3037] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.930604 29007 raft_consensus.cc:3037] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:14.933598 28876 raft_consensus.cc:2454] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:14.933598 29007 raft_consensus.cc:2454] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:14.934113 29071 leader_election.cc:302] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:14.934470 29811 raft_consensus.cc:2781] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:14.934960 29811 raft_consensus.cc:686] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:14.935375 29811 consensus_queue.cc:237] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:14.938310 28598 catalog_manager.cc:5484] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:15.124191 29811 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:15.131309 29855 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:15.212577 29853 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:15.213061 29839 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:15.326889 29857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:15.327230 29857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:15.335088 29857 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:02:15.410812 29855 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:15.416146 29811 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 232.944 ms
time per row: 0.116472 ms
I20240501 14:02:15.739993 28598 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:15.749516 29118 tablet_service.cc:1467] Processing CreateTablet for tablet 6feda9b53e484e7ea40a5890b7d51ac4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.749619 28723 tablet_service.cc:1467] Processing CreateTablet for tablet 6feda9b53e484e7ea40a5890b7d51ac4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.750041 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6feda9b53e484e7ea40a5890b7d51ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.750082 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6feda9b53e484e7ea40a5890b7d51ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.750247 28855 tablet_service.cc:1467] Processing CreateTablet for tablet 6feda9b53e484e7ea40a5890b7d51ac4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.750675 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6feda9b53e484e7ea40a5890b7d51ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.750480 28985 tablet_service.cc:1467] Processing CreateTablet for tablet e0bf08cbb6d347d783b1fa7762c802df (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.751390 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bf08cbb6d347d783b1fa7762c802df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.754247 28725 tablet_service.cc:1467] Processing CreateTablet for tablet e0bf08cbb6d347d783b1fa7762c802df (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.754652 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bf08cbb6d347d783b1fa7762c802df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.756749 28856 tablet_service.cc:1467] Processing CreateTablet for tablet 014635e01a22420b8bc21e879449749b (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.757115 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014635e01a22420b8bc21e879449749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.758999 28724 tablet_service.cc:1467] Processing CreateTablet for tablet 014635e01a22420b8bc21e879449749b (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.759389 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014635e01a22420b8bc21e879449749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.763672 29117 tablet_service.cc:1467] Processing CreateTablet for tablet e0bf08cbb6d347d783b1fa7762c802df (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.764117 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bf08cbb6d347d783b1fa7762c802df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.767633 29921 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:15.767858 29116 tablet_service.cc:1467] Processing CreateTablet for tablet 014635e01a22420b8bc21e879449749b (DEFAULT_TABLE table=rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:15.768488 29116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014635e01a22420b8bc21e879449749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:15.770165 29922 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:15.771319 29921 tablet_bootstrap.cc:654] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.772850 29922 tablet_bootstrap.cc:654] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.781257 29922 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:15.781584 29922 ts_tablet_manager.cc:1367] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 14:02:15.782578 29922 raft_consensus.cc:348] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.782949 29922 raft_consensus.cc:374] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.783098 29922 raft_consensus.cc:729] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.783496 29922 consensus_queue.cc:260] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.787142 29922 ts_tablet_manager.cc:1398] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:02:15.788414 29922 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:15.790257 29922 tablet_bootstrap.cc:654] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.806181 29925 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:15.811543 29925 tablet_bootstrap.cc:654] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.813673 29920 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:15.815501 29921 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:15.815708 29921 ts_tablet_manager.cc:1367] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.029s
I20240501 14:02:15.816696 29920 tablet_bootstrap.cc:654] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.816579 29921 raft_consensus.cc:348] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.816813 29921 raft_consensus.cc:374] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.816923 29921 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.817219 29921 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.819355 29921 ts_tablet_manager.cc:1398] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:02:15.819360 29922 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:15.819562 29922 ts_tablet_manager.cc:1367] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.022s
I20240501 14:02:15.819700 29925 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:15.819900 29925 ts_tablet_manager.cc:1367] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 14:02:15.820530 29920 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:15.820714 29920 ts_tablet_manager.cc:1367] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:02:15.820577 29922 raft_consensus.cc:348] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.820828 29922 raft_consensus.cc:374] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.820791 29925 raft_consensus.cc:348] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.820964 29922 raft_consensus.cc:729] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.821054 29925 raft_consensus.cc:374] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.821220 29925 raft_consensus.cc:729] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.821349 29922 consensus_queue.cc:260] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.821566 29925 consensus_queue.cc:260] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.822141 29922 ts_tablet_manager.cc:1398] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:15.822515 29920 raft_consensus.cc:348] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.822922 29920 raft_consensus.cc:374] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.823122 29920 raft_consensus.cc:729] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.823588 29920 consensus_queue.cc:260] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.823932 29925 ts_tablet_manager.cc:1398] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:02:15.824270 29925 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:15.824878 29920 ts_tablet_manager.cc:1398] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:15.825516 29920 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:15.826201 29925 tablet_bootstrap.cc:654] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.827221 29920 tablet_bootstrap.cc:654] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.829613 29925 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:15.829648 29920 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:15.829838 29925 ts_tablet_manager.cc:1367] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 14:02:15.829838 29920 ts_tablet_manager.cc:1367] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:15.830479 29920 raft_consensus.cc:348] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.830687 29920 raft_consensus.cc:374] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.830602 29925 raft_consensus.cc:348] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.830773 29920 raft_consensus.cc:729] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.830883 29925 raft_consensus.cc:374] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.831022 29925 raft_consensus.cc:729] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.831027 29920 consensus_queue.cc:260] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.831509 29920 ts_tablet_manager.cc:1398] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:15.831509 29925 consensus_queue.cc:260] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.831825 29920 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:15.832044 29925 ts_tablet_manager.cc:1398] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:02:15.832369 29925 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:15.833417 29920 tablet_bootstrap.cc:654] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.834878 29925 tablet_bootstrap.cc:654] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:15.838011 29920 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:15.838223 29920 ts_tablet_manager.cc:1367] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20240501 14:02:15.839043 29920 raft_consensus.cc:348] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.839217 29920 raft_consensus.cc:374] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.839326 29920 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.839555 29920 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.839865 29925 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:15.840009 29925 ts_tablet_manager.cc:1367] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 14:02:15.840071 29920 ts_tablet_manager.cc:1398] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:15.840730 29925 raft_consensus.cc:348] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.840956 29925 raft_consensus.cc:374] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:15.841065 29925 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:15.841352 29925 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.841921 29925 ts_tablet_manager.cc:1398] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:15.851876 29929 raft_consensus.cc:482] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:15.852072 29929 raft_consensus.cc:504] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.853725 29929 leader_election.cc:288] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:15.854804 29929 raft_consensus.cc:482] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:15.854991 29929 raft_consensus.cc:504] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.855942 29929 leader_election.cc:288] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:15.863451 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014635e01a22420b8bc21e879449749b" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:15.863801 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014635e01a22420b8bc21e879449749b" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:15.864004 29138 raft_consensus.cc:2454] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:15.864050 28876 raft_consensus.cc:2454] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:15.863487 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:15.864591 29137 raft_consensus.cc:2454] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:15.864681 28678 leader_election.cc:302] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:15.865319 29929 raft_consensus.cc:2781] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:15.865437 29929 raft_consensus.cc:482] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:15.865546 29929 raft_consensus.cc:3037] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.865644 28678 leader_election.cc:302] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:15.866742 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:15.866993 29005 raft_consensus.cc:2454] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:15.867586 29938 raft_consensus.cc:2781] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:15.867694 29938 raft_consensus.cc:482] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:15.867828 29938 raft_consensus.cc:3037] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.868580 29929 raft_consensus.cc:504] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.869308 29929 leader_election.cc:288] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:15.869750 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014635e01a22420b8bc21e879449749b" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:15.869937 29137 raft_consensus.cc:3037] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.869936 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014635e01a22420b8bc21e879449749b" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:15.870185 28876 raft_consensus.cc:3037] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.870178 29938 raft_consensus.cc:504] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.871326 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:15.871537 29138 raft_consensus.cc:3037] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.872478 29137 raft_consensus.cc:2454] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:15.872907 28678 leader_election.cc:302] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:15.873160 28876 raft_consensus.cc:2454] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:15.873185 29929 raft_consensus.cc:2781] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:15.874243 29138 raft_consensus.cc:2454] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:15.874660 28678 leader_election.cc:302] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:15.875377 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:15.875608 29005 raft_consensus.cc:3037] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.875712 29940 raft_consensus.cc:2781] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:15.875918 29940 raft_consensus.cc:686] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Leader. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:02:15.876709 29940 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:15.877602 29005 raft_consensus.cc:2454] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:15.878129 29938 leader_election.cc:288] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:15.878882 29929 raft_consensus.cc:686] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Leader. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:02:15.879215 29929 consensus_queue.cc:237] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.882222 28602 catalog_manager.cc:5484] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a85ba3b514663af5320cd14a9e144 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:15.889405 28602 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a85ba3b514663af5320cd14a9e144 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:15.952385 29795 raft_consensus.cc:482] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:15.952626 29795 raft_consensus.cc:504] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.953576 29795 leader_election.cc:288] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:15.954377 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6feda9b53e484e7ea40a5890b7d51ac4" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:15.954739 28743 raft_consensus.cc:2454] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:15.955307 28811 leader_election.cc:302] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:15.955655 29795 raft_consensus.cc:2781] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:15.955804 29795 raft_consensus.cc:482] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:15.955936 29795 raft_consensus.cc:3037] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.958307 29795 raft_consensus.cc:504] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.959156 29795 leader_election.cc:288] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:15.959656 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6feda9b53e484e7ea40a5890b7d51ac4" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:15.959884 28743 raft_consensus.cc:3037] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:15.961310 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6feda9b53e484e7ea40a5890b7d51ac4" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:15.961403 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6feda9b53e484e7ea40a5890b7d51ac4" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:15.961582 29138 raft_consensus.cc:2454] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:15.962493 28743 raft_consensus.cc:2454] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:15.962908 28811 leader_election.cc:302] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters: 3a3972b170c54344a4ba61f176f837ba
I20240501 14:02:15.963196 29795 raft_consensus.cc:2781] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:15.963634 29795 raft_consensus.cc:686] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:02:15.963932 29795 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:15.966709 28602 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:16.284632 29929 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:16.291003 29929 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:16.352741 29795 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:16.357436 29137 raft_consensus.cc:3037] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:16.360879 29944 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:16.364755 29946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:16.365202 29946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:16.373541 29946 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:02:16.439116 29940 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:16.455842 29929 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 240.563 ms
time per row: 0.120282 ms
I20240501 14:02:16.772634 28598 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:16.781984 28724 tablet_service.cc:1467] Processing CreateTablet for tablet 2998b60071cd408aa1cd06fb57d76a8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.782330 28856 tablet_service.cc:1467] Processing CreateTablet for tablet 2998b60071cd408aa1cd06fb57d76a8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.782528 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998b60071cd408aa1cd06fb57d76a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.782467 28985 tablet_service.cc:1467] Processing CreateTablet for tablet 2998b60071cd408aa1cd06fb57d76a8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.782790 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998b60071cd408aa1cd06fb57d76a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.782989 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998b60071cd408aa1cd06fb57d76a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.783316 28986 tablet_service.cc:1467] Processing CreateTablet for tablet e9bb3eaf3f0e40f8902b9426a845d9b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.783702 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bb3eaf3f0e40f8902b9426a845d9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.790747 29118 tablet_service.cc:1467] Processing CreateTablet for tablet e9bb3eaf3f0e40f8902b9426a845d9b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.791383 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bb3eaf3f0e40f8902b9426a845d9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.791405 28725 tablet_service.cc:1467] Processing CreateTablet for tablet e9bb3eaf3f0e40f8902b9426a845d9b2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.791932 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bb3eaf3f0e40f8902b9426a845d9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.792922 30010 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:16.792912 28855 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.793375 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7f2a8d8a84faf86c4de16b3cc6663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.794723 30010 tablet_bootstrap.cc:654] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.795286 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.795742 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7f2a8d8a84faf86c4de16b3cc6663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.805176 30008 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:16.806742 28985 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 (DEFAULT_TABLE table=rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:16.807145 30008 tablet_bootstrap.cc:654] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.807312 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7f2a8d8a84faf86c4de16b3cc6663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:16.810101 30012 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:16.813557 30012 tablet_bootstrap.cc:654] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.817365 30010 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:16.817395 30009 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:16.817575 30010 ts_tablet_manager.cc:1367] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.014s
I20240501 14:02:16.818745 30010 raft_consensus.cc:348] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:16.819108 30010 raft_consensus.cc:374] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.819214 30009 tablet_bootstrap.cc:654] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.819298 30010 raft_consensus.cc:729] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.819639 30010 consensus_queue.cc:260] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:16.820399 30010 ts_tablet_manager.cc:1398] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:16.820737 30010 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:16.820912 30008 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:16.821892 30012 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:16.822094 30012 ts_tablet_manager.cc:1367] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240501 14:02:16.822451 30010 tablet_bootstrap.cc:654] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.822547 30008 ts_tablet_manager.cc:1367] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20240501 14:02:16.822960 30012 raft_consensus.cc:348] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:16.823237 30012 raft_consensus.cc:374] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.823525 30012 raft_consensus.cc:729] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.823542 30008 raft_consensus.cc:348] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.823911 30012 consensus_queue.cc:260] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:16.824627 30012 ts_tablet_manager.cc:1398] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:16.824949 30012 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:16.825394 30008 raft_consensus.cc:374] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.825585 30008 raft_consensus.cc:729] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.825922 30008 consensus_queue.cc:260] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.826570 30012 tablet_bootstrap.cc:654] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.826756 30009 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:16.826969 30009 ts_tablet_manager.cc:1367] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:02:16.827525 29944 raft_consensus.cc:482] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:16.827754 29944 raft_consensus.cc:504] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.827929 30009 raft_consensus.cc:348] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.828155 30009 raft_consensus.cc:374] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.828260 30009 raft_consensus.cc:729] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.828521 30009 consensus_queue.cc:260] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.828750 29944 leader_election.cc:288] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:16.829411 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998b60071cd408aa1cd06fb57d76a8f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:16.829598 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998b60071cd408aa1cd06fb57d76a8f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:16.830011 28743 raft_consensus.cc:2454] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:16.830334 30009 ts_tablet_manager.cc:1398] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:16.830541 28811 leader_election.cc:302] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:16.831843 30012 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
W20240501 14:02:16.831848 28810 leader_election.cc:341] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:16.832060 30012 ts_tablet_manager.cc:1367] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:02:16.832424 30010 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:16.832412 29944 raft_consensus.cc:2781] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:16.832520 29944 raft_consensus.cc:482] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:16.832633 29944 raft_consensus.cc:3037] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:16.832760 30010 ts_tablet_manager.cc:1367] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240501 14:02:16.832901 30012 raft_consensus.cc:348] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.833133 30012 raft_consensus.cc:374] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.833226 30012 raft_consensus.cc:729] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.833488 30012 consensus_queue.cc:260] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.833766 30009 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:16.833907 30010 raft_consensus.cc:348] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.834192 30010 raft_consensus.cc:374] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.834324 30010 raft_consensus.cc:729] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.834669 30010 consensus_queue.cc:260] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.835482 30010 ts_tablet_manager.cc:1398] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:16.835510 29944 raft_consensus.cc:504] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.835767 30009 tablet_bootstrap.cc:654] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.835856 30010 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:16.837642 30010 tablet_bootstrap.cc:654] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.839365 30012 ts_tablet_manager.cc:1398] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:02:16.840586 30009 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:16.841451 30009 ts_tablet_manager.cc:1367] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 14:02:16.841385 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998b60071cd408aa1cd06fb57d76a8f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:16.841674 28743 raft_consensus.cc:3037] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:16.841755 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2998b60071cd408aa1cd06fb57d76a8f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:16.842041 29007 raft_consensus.cc:3037] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:16.844267 28743 raft_consensus.cc:2454] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:16.844794 28811 leader_election.cc:302] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:16.845117 29996 raft_consensus.cc:2781] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:16.845290 29996 raft_consensus.cc:686] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:02:16.845645 29996 consensus_queue.cc:237] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:16.846025 29007 raft_consensus.cc:2454] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:16.846464 29944 leader_election.cc:288] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Requested vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:16.847355 30008 ts_tablet_manager.cc:1398] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20240501 14:02:16.847887 30008 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:16.847739 30009 raft_consensus.cc:348] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:16.848109 30009 raft_consensus.cc:374] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.848246 30009 raft_consensus.cc:729] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.848652 30009 consensus_queue.cc:260] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:16.852589 30008 tablet_bootstrap.cc:654] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:16.852985 30009 ts_tablet_manager.cc:1398] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20240501 14:02:16.856535 30010 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:16.856859 30010 ts_tablet_manager.cc:1367] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.006s
I20240501 14:02:16.857424 30008 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:16.857614 30008 ts_tablet_manager.cc:1367] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240501 14:02:16.857820 30010 raft_consensus.cc:348] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.858000 30010 raft_consensus.cc:374] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.858127 30010 raft_consensus.cc:729] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.858361 30010 consensus_queue.cc:260] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.858173 28600 catalog_manager.cc:5484] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:16.858423 30008 raft_consensus.cc:348] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.858732 30008 raft_consensus.cc:374] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:16.858824 30008 raft_consensus.cc:729] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:16.859066 30010 ts_tablet_manager.cc:1398] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:16.859067 30008 consensus_queue.cc:260] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.859608 30008 ts_tablet_manager.cc:1398] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:02:16.883136 29996 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:16.883358 29996 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.883992 29996 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:16.884599 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:16.884632 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:16.884824 29007 raft_consensus.cc:2454] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:16.884929 29137 raft_consensus.cc:2454] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:16.885324 28810 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:16.885638 29996 raft_consensus.cc:2781] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:16.885744 29996 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:16.885861 29996 raft_consensus.cc:3037] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:16.887913 29996 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.888592 29996 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:16.889561 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:16.889623 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:16.889844 29137 raft_consensus.cc:3037] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:16.889876 29007 raft_consensus.cc:3037] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:16.892107 29007 raft_consensus.cc:2454] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:16.892160 29137 raft_consensus.cc:2454] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:16.892580 28810 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:16.892858 29996 raft_consensus.cc:2781] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:16.893033 29996 raft_consensus.cc:686] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:02:16.893384 29996 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:16.896116 28597 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:17.162381 29909 raft_consensus.cc:482] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:17.162586 29909 raft_consensus.cc:504] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:17.163442 29909 leader_election.cc:288] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:17.164167 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bb3eaf3f0e40f8902b9426a845d9b2" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:17.164459 28743 raft_consensus.cc:2454] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:17.165012 28942 leader_election.cc:302] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:17.165360 29909 raft_consensus.cc:2781] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:17.165526 29909 raft_consensus.cc:482] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:17.165639 29909 raft_consensus.cc:3037] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:17.169294 29909 raft_consensus.cc:504] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:17.170456 29909 leader_election.cc:288] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:17.170845 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bb3eaf3f0e40f8902b9426a845d9b2" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:17.171095 28743 raft_consensus.cc:3037] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:17.174000 28743 raft_consensus.cc:2454] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:17.174526 28942 leader_election.cc:302] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:17.174898 29909 raft_consensus.cc:2781] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:17.175341 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bb3eaf3f0e40f8902b9426a845d9b2" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:17.175369 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bb3eaf3f0e40f8902b9426a845d9b2" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:17.175606 29138 raft_consensus.cc:3037] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:17.178416 29138 raft_consensus.cc:2454] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:17.179761 29909 raft_consensus.cc:686] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:17.180150 29909 consensus_queue.cc:237] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:17.183322 28597 catalog_manager.cc:5484] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:02:17.285362 29996 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:17.292286 29944 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:17.374590 29944 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:17.379397 29944 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:17.598345 29909 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:17.604544 30038 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:17.641728 30027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:17.642092 30027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:17.651531 30027 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 247.635 ms
time per row: 0.123817 ms
I20240501 14:02:18.060729 28600 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:18.070326 28856 tablet_service.cc:1467] Processing CreateTablet for tablet 12510f71b6a742ab8cc359243de8119b (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.070920 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12510f71b6a742ab8cc359243de8119b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.070847 28985 tablet_service.cc:1467] Processing CreateTablet for tablet 12510f71b6a742ab8cc359243de8119b (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.071302 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12510f71b6a742ab8cc359243de8119b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.071542 28725 tablet_service.cc:1467] Processing CreateTablet for tablet de161b10c84f4bcc85f8cf7b199f99db (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.071967 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de161b10c84f4bcc85f8cf7b199f99db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.074187 28724 tablet_service.cc:1467] Processing CreateTablet for tablet 12510f71b6a742ab8cc359243de8119b (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.074611 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12510f71b6a742ab8cc359243de8119b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.075493 28986 tablet_service.cc:1467] Processing CreateTablet for tablet 39f1ab3a94db41b08ac90bba52418a93 (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.075877 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1ab3a94db41b08ac90bba52418a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.079218 28723 tablet_service.cc:1467] Processing CreateTablet for tablet 39f1ab3a94db41b08ac90bba52418a93 (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.079638 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1ab3a94db41b08ac90bba52418a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.079603 28855 tablet_service.cc:1467] Processing CreateTablet for tablet de161b10c84f4bcc85f8cf7b199f99db (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.080132 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de161b10c84f4bcc85f8cf7b199f99db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.082193 29118 tablet_service.cc:1467] Processing CreateTablet for tablet de161b10c84f4bcc85f8cf7b199f99db (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.082821 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de161b10c84f4bcc85f8cf7b199f99db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.084194 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 39f1ab3a94db41b08ac90bba52418a93 (DEFAULT_TABLE table=rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:18.084594 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1ab3a94db41b08ac90bba52418a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:18.082484 30092 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:18.086977 30092 tablet_bootstrap.cc:654] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.093039 30092 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:18.094985 30093 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:18.095314 30092 ts_tablet_manager.cc:1367] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 14:02:18.096196 30092 raft_consensus.cc:348] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.096426 30092 raft_consensus.cc:374] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.096909 30093 tablet_bootstrap.cc:654] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.097877 30092 raft_consensus.cc:729] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.098435 30092 consensus_queue.cc:260] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.098768 30090 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:18.099542 30092 ts_tablet_manager.cc:1398] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:18.099619 30091 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:18.100184 30092 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:18.100713 30090 tablet_bootstrap.cc:654] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.101883 30091 tablet_bootstrap.cc:654] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.102133 30092 tablet_bootstrap.cc:654] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.108762 30038 raft_consensus.cc:482] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:18.109006 30038 raft_consensus.cc:504] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.109938 30038 leader_election.cc:288] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:18.110594 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:18.110803 30093 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:18.110788 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:18.110957 30093 ts_tablet_manager.cc:1367] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
W20240501 14:02:18.111553 28940 leader_election.cc:341] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:02:18.111886 28942 leader_election.cc:341] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:18.111977 30091 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:18.112043 28942 leader_election.cc:302] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8; no voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba
I20240501 14:02:18.112141 30091 ts_tablet_manager.cc:1367] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 14:02:18.112872 30038 raft_consensus.cc:2726] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:18.113029 30091 raft_consensus.cc:348] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.113262 30091 raft_consensus.cc:374] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.113375 30091 raft_consensus.cc:729] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.113282 30093 raft_consensus.cc:348] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.113508 30093 raft_consensus.cc:374] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.113616 30093 raft_consensus.cc:729] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.113687 30091 consensus_queue.cc:260] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.113873 30093 consensus_queue.cc:260] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.114446 30093 ts_tablet_manager.cc:1398] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:02:18.114836 30093 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:18.116708 30093 tablet_bootstrap.cc:654] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.120358 30090 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:18.120555 30090 ts_tablet_manager.cc:1367] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s
I20240501 14:02:18.121325 30090 raft_consensus.cc:348] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.121551 30090 raft_consensus.cc:374] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.121654 30090 raft_consensus.cc:729] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.121919 30090 consensus_queue.cc:260] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.122552 30090 ts_tablet_manager.cc:1398] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:18.123287 30090 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:18.123504 30091 ts_tablet_manager.cc:1398] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:02:18.125051 30090 tablet_bootstrap.cc:654] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.127982 30091 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:18.130656 30091 tablet_bootstrap.cc:654] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.131206 30092 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:18.131512 30092 ts_tablet_manager.cc:1367] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.005s
I20240501 14:02:18.132691 30092 raft_consensus.cc:348] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.133220 30093 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:18.133265 30092 raft_consensus.cc:374] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.133461 30093 ts_tablet_manager.cc:1367] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20240501 14:02:18.133473 30092 raft_consensus.cc:729] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.134267 29906 raft_consensus.cc:482] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:18.134488 29906 raft_consensus.cc:504] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.134667 30093 raft_consensus.cc:348] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.134994 30093 raft_consensus.cc:374] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.135103 30093 raft_consensus.cc:729] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.135658 29906 leader_election.cc:288] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:18.136075 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:18.136314 30092 consensus_queue.cc:260] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.136497 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:18.136783 28744 raft_consensus.cc:2454] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
W20240501 14:02:18.136940 29071 leader_election.cc:341] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:18.137305 29073 leader_election.cc:302] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters: cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:02:18.137753 29906 raft_consensus.cc:2781] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:18.137902 29906 raft_consensus.cc:482] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:18.138032 29906 raft_consensus.cc:3037] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.139230 30091 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:18.139405 30091 ts_tablet_manager.cc:1367] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240501 14:02:18.140182 30091 raft_consensus.cc:348] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.140417 30091 raft_consensus.cc:374] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.140486 30091 raft_consensus.cc:729] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.140796 30091 consensus_queue.cc:260] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.141098 29906 raft_consensus.cc:504] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.142148 30092 ts_tablet_manager.cc:1398] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:02:18.142304 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
W20240501 14:02:18.143000 29071 leader_election.cc:341] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:18.143342 29906 leader_election.cc:288] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:18.137900 30093 consensus_queue.cc:260] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.144455 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:18.144716 28744 raft_consensus.cc:3037] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.147266 30091 ts_tablet_manager.cc:1398] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:02:18.147660 30091 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:18.147945 30090 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:18.148186 30090 ts_tablet_manager.cc:1367] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.008s
I20240501 14:02:18.149154 30090 raft_consensus.cc:348] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.149525 30091 tablet_bootstrap.cc:654] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:18.149617 30090 raft_consensus.cc:374] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.152410 30093 ts_tablet_manager.cc:1398] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.019s user 0.000s sys 0.003s
I20240501 14:02:18.152762 28744 raft_consensus.cc:2454] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:18.153246 29073 leader_election.cc:302] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters: cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:02:18.159897 29906 raft_consensus.cc:2781] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.160084 29906 raft_consensus.cc:686] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:18.160439 29906 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.149751 30090 raft_consensus.cc:729] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.171262 30090 consensus_queue.cc:260] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.173406 30090 ts_tablet_manager.cc:1398] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20240501 14:02:18.179268 28597 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:02:18.182207 30091 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:18.182385 30091 ts_tablet_manager.cc:1367] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.017s
I20240501 14:02:18.183156 30091 raft_consensus.cc:348] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.183375 30091 raft_consensus.cc:374] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:18.183458 30091 raft_consensus.cc:729] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:18.183686 30091 consensus_queue.cc:260] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.184229 30091 ts_tablet_manager.cc:1398] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:18.199743 30038 raft_consensus.cc:482] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:18.199956 30038 raft_consensus.cc:504] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.200531 30038 leader_election.cc:288] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:18.201138 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12510f71b6a742ab8cc359243de8119b" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:18.201136 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12510f71b6a742ab8cc359243de8119b" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:18.201375 28744 raft_consensus.cc:2454] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:18.201393 28876 raft_consensus.cc:2454] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:18.201848 28940 leader_election.cc:302] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:18.202126 30038 raft_consensus.cc:2781] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:18.202214 30038 raft_consensus.cc:482] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:18.202322 30038 raft_consensus.cc:3037] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.204325 30038 raft_consensus.cc:504] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.205052 30038 leader_election.cc:288] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:18.205531 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12510f71b6a742ab8cc359243de8119b" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:18.205709 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12510f71b6a742ab8cc359243de8119b" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:18.205785 28876 raft_consensus.cc:3037] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.205909 28744 raft_consensus.cc:3037] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.207837 28876 raft_consensus.cc:2454] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:18.207870 28744 raft_consensus.cc:2454] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:18.208271 28942 leader_election.cc:302] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:18.208657 30038 raft_consensus.cc:2781] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.208959 30038 raft_consensus.cc:686] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:18.209268 30038 consensus_queue.cc:237] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:18.212507 28597 catalog_manager.cc:5484] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:18.445762 29906 raft_consensus.cc:482] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:18.445964 29906 raft_consensus.cc:504] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.446574 29906 leader_election.cc:288] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:18.447302 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:18.447304 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:18.447607 28744 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:18.447607 29007 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:18.448155 29073 leader_election.cc:302] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:18.448462 29906 raft_consensus.cc:2781] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:18.448568 29906 raft_consensus.cc:482] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:18.448652 29906 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.450446 29906 raft_consensus.cc:504] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.451064 29906 leader_election.cc:288] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:18.451601 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:18.451601 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:18.451828 29007 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.451850 28744 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.454708 28744 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:18.454708 29007 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:18.455199 29072 leader_election.cc:302] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:18.455520 29906 raft_consensus.cc:2781] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:18.455708 29906 raft_consensus.cc:686] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:18.455969 29906 consensus_queue.cc:237] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:18.459149 28597 catalog_manager.cc:5484] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:18.513870 30034 raft_consensus.cc:482] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:18.514096 30034 raft_consensus.cc:504] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:18.514894 30034 leader_election.cc:288] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:18.515530 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:18.515516 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:18.515841 28744 raft_consensus.cc:2379] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd3c04ebb6c84b15a1b3adc748737094 in current term 1: Already voted for candidate 3a3972b170c54344a4ba61f176f837ba in this term.
I20240501 14:02:18.516309 28811 leader_election.cc:302] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba
I20240501 14:02:18.516610 30034 raft_consensus.cc:3037] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:18.519318 30034 raft_consensus.cc:2726] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:02:18.543329 29906 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:18.558305 30104 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:18.614699 30038 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:18.624074 30025 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:18.876320 30109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:18.876680 30109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:18.884549 30109 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:02:18.973763 30104 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:18.984319 29906 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 267.265 ms
time per row: 0.133633 ms
I20240501 14:02:19.319252 28597 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:19.328907 28856 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0bc741ed064dd7ad8ec70219496c6f (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.329334 28985 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0bc741ed064dd7ad8ec70219496c6f (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.329493 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0bc741ed064dd7ad8ec70219496c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.329859 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0bc741ed064dd7ad8ec70219496c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.330518 28725 tablet_service.cc:1467] Processing CreateTablet for tablet 6aaf52ee71b44511b7b8909711ab0a42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.330994 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aaf52ee71b44511b7b8909711ab0a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.331842 28987 tablet_service.cc:1467] Processing CreateTablet for tablet 6aaf52ee71b44511b7b8909711ab0a42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.332264 28987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aaf52ee71b44511b7b8909711ab0a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.335170 28986 tablet_service.cc:1467] Processing CreateTablet for tablet f4402de0b48041c383568feb9365147e (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.335639 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4402de0b48041c383568feb9365147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.339823 29117 tablet_service.cc:1467] Processing CreateTablet for tablet f4402de0b48041c383568feb9365147e (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.340327 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4402de0b48041c383568feb9365147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.341019 28854 tablet_service.cc:1467] Processing CreateTablet for tablet 6aaf52ee71b44511b7b8909711ab0a42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.341054 28855 tablet_service.cc:1467] Processing CreateTablet for tablet f4402de0b48041c383568feb9365147e (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.341473 28854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aaf52ee71b44511b7b8909711ab0a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.341614 29118 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0bc741ed064dd7ad8ec70219496c6f (DEFAULT_TABLE table=rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:19.342185 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0bc741ed064dd7ad8ec70219496c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.345992 30172 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:19.346686 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4402de0b48041c383568feb9365147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:19.348265 30172 tablet_bootstrap.cc:654] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.354482 30173 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:19.357492 30173 tablet_bootstrap.cc:654] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.365660 30175 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:19.368578 30175 tablet_bootstrap.cc:654] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.370127 30171 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:19.371759 30173 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:19.372023 30173 ts_tablet_manager.cc:1367] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240501 14:02:19.372988 30172 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:19.373039 30173 raft_consensus.cc:348] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.373241 30172 ts_tablet_manager.cc:1367] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.011s
I20240501 14:02:19.373325 30173 raft_consensus.cc:374] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.373420 30173 raft_consensus.cc:729] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.373672 30173 consensus_queue.cc:260] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.373782 30171 tablet_bootstrap.cc:654] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.374579 30172 raft_consensus.cc:348] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.375221 30173 ts_tablet_manager.cc:1398] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:19.375507 30173 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:19.376380 30175 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:19.376426 30171 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:19.376559 30175 ts_tablet_manager.cc:1367] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 14:02:19.376570 30171 ts_tablet_manager.cc:1367] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 14:02:19.377187 30173 tablet_bootstrap.cc:654] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.377351 30171 raft_consensus.cc:348] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.377385 30175 raft_consensus.cc:348] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.377575 30175 raft_consensus.cc:374] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.377574 30171 raft_consensus.cc:374] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.377664 30175 raft_consensus.cc:729] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.377663 30171 raft_consensus.cc:729] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.377902 30171 consensus_queue.cc:260] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.377924 30175 consensus_queue.cc:260] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.378679 30175 ts_tablet_manager.cc:1398] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:02:19.379021 30175 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:19.380764 30175 tablet_bootstrap.cc:654] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.380903 30172 raft_consensus.cc:374] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.381623 30172 raft_consensus.cc:729] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.382037 30172 consensus_queue.cc:260] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.383033 30172 ts_tablet_manager.cc:1398] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:02:19.388294 30171 ts_tablet_manager.cc:1398] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240501 14:02:19.388765 30171 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:19.390983 30171 tablet_bootstrap.cc:654] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.393803 30171 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:19.393909 30175 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:19.393985 30173 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:19.393996 30171 ts_tablet_manager.cc:1367] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 14:02:19.394179 30175 ts_tablet_manager.cc:1367] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240501 14:02:19.394178 30173 ts_tablet_manager.cc:1367] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20240501 14:02:19.394901 30175 raft_consensus.cc:348] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.394882 30171 raft_consensus.cc:348] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.395094 30175 raft_consensus.cc:374] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.395118 30171 raft_consensus.cc:374] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.395169 30175 raft_consensus.cc:729] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.395187 30171 raft_consensus.cc:729] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.395071 30173 raft_consensus.cc:348] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.395380 30173 raft_consensus.cc:374] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.395484 30173 raft_consensus.cc:729] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.395437 30171 consensus_queue.cc:260] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.395455 30175 consensus_queue.cc:260] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.395752 30173 consensus_queue.cc:260] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.396127 30171 ts_tablet_manager.cc:1398] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:19.396405 30173 ts_tablet_manager.cc:1398] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:19.396745 30173 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:19.397126 30175 ts_tablet_manager.cc:1398] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:19.398500 30171 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:19.398514 30173 tablet_bootstrap.cc:654] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.400650 30171 tablet_bootstrap.cc:654] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:19.401253 29988 raft_consensus.cc:482] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:19.401466 29988 raft_consensus.cc:504] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.402468 29988 leader_election.cc:288] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:19.403023 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aaf52ee71b44511b7b8909711ab0a42" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:19.403378 29005 raft_consensus.cc:2454] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:19.403537 30173 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:19.403728 30173 ts_tablet_manager.cc:1367] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:02:19.404379 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aaf52ee71b44511b7b8909711ab0a42" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:19.404703 28876 raft_consensus.cc:2454] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:19.404867 30173 raft_consensus.cc:348] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.405144 30173 raft_consensus.cc:374] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.405274 30173 raft_consensus.cc:729] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.405604 30173 consensus_queue.cc:260] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.406513 30173 ts_tablet_manager.cc:1398] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:02:19.407287 28679 leader_election.cc:302] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:19.407706 29988 raft_consensus.cc:2781] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:19.407845 29988 raft_consensus.cc:482] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:19.407945 29988 raft_consensus.cc:3037] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.407966 30171 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:19.408114 30171 ts_tablet_manager.cc:1367] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 14:02:19.408931 30171 raft_consensus.cc:348] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.409168 30171 raft_consensus.cc:374] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:19.409252 30171 raft_consensus.cc:729] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:19.409494 30171 consensus_queue.cc:260] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.410070 30171 ts_tablet_manager.cc:1398] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:19.410810 29988 raft_consensus.cc:504] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.411573 29988 leader_election.cc:288] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:19.412868 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aaf52ee71b44511b7b8909711ab0a42" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:19.412935 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aaf52ee71b44511b7b8909711ab0a42" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:19.413118 29005 raft_consensus.cc:3037] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.413134 28876 raft_consensus.cc:3037] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.416087 29005 raft_consensus.cc:2454] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:19.416108 28876 raft_consensus.cc:2454] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:19.416589 28679 leader_election.cc:302] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:19.417062 29988 raft_consensus.cc:2781] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:19.417553 29988 raft_consensus.cc:686] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Leader. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:02:19.417939 29988 consensus_queue.cc:237] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:19.421208 28603 catalog_manager.cc:5484] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a85ba3b514663af5320cd14a9e144 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:19.428305 30182 raft_consensus.cc:482] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:19.428558 30182 raft_consensus.cc:504] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.429345 30182 leader_election.cc:288] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:19.429864 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0bc741ed064dd7ad8ec70219496c6f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:19.429883 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0bc741ed064dd7ad8ec70219496c6f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:19.430078 29005 raft_consensus.cc:2454] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:19.430094 29138 raft_consensus.cc:2454] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:19.430510 28810 leader_election.cc:302] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:19.430831 30182 raft_consensus.cc:2781] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:19.430958 30182 raft_consensus.cc:482] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:19.431032 30182 raft_consensus.cc:3037] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.433039 30182 raft_consensus.cc:504] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.433655 30182 leader_election.cc:288] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:19.434262 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0bc741ed064dd7ad8ec70219496c6f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:19.434235 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0bc741ed064dd7ad8ec70219496c6f" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:19.434463 29138 raft_consensus.cc:3037] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.434476 29005 raft_consensus.cc:3037] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.436439 29138 raft_consensus.cc:2454] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:19.436453 29005 raft_consensus.cc:2454] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:19.436933 28809 leader_election.cc:302] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:19.437314 30182 raft_consensus.cc:2781] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:19.437501 30182 raft_consensus.cc:686] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:02:19.437820 30182 consensus_queue.cc:237] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:19.440878 28603 catalog_manager.cc:5484] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:19.519829 29906 raft_consensus.cc:482] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:19.520031 29906 raft_consensus.cc:504] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.520725 29906 leader_election.cc:288] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:19.521293 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4402de0b48041c383568feb9365147e" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:19.521306 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4402de0b48041c383568feb9365147e" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:19.521595 29005 raft_consensus.cc:2454] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:19.521606 28876 raft_consensus.cc:2454] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:19.522145 29071 leader_election.cc:302] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:19.522454 29906 raft_consensus.cc:2781] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:19.522585 29906 raft_consensus.cc:482] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:19.522704 29906 raft_consensus.cc:3037] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.524713 29906 raft_consensus.cc:504] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.525306 29906 leader_election.cc:288] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:19.525903 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4402de0b48041c383568feb9365147e" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:19.525949 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4402de0b48041c383568feb9365147e" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:19.526157 29005 raft_consensus.cc:3037] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.526363 28876 raft_consensus.cc:3037] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:19.528375 29005 raft_consensus.cc:2454] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:19.528743 28876 raft_consensus.cc:2454] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:19.528774 29072 leader_election.cc:302] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:19.529035 29906 raft_consensus.cc:2781] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:19.529234 29906 raft_consensus.cc:686] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:19.529515 29906 consensus_queue.cc:237] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:19.532436 28603 catalog_manager.cc:5484] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:19.829492 30182 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:19.834898 30110 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:19.919855 30189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:19.920231 30189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:19.928586 30189 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:19.937417 29988 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:19.949936 30200 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:19.970489 30104 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:19.975494 29906 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 298.521 ms
time per row: 0.149261 ms
I20240501 14:02:20.411024 28598 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:20.420631 28725 tablet_service.cc:1467] Processing CreateTablet for tablet 135da52054584a27959efdd9126cfcef (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.421165 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135da52054584a27959efdd9126cfcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.421655 28723 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6a54ac32694640bffdf1c7f3b035cc (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.422057 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6a54ac32694640bffdf1c7f3b035cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.422362 28985 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6a54ac32694640bffdf1c7f3b035cc (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.422827 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6a54ac32694640bffdf1c7f3b035cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.423128 29118 tablet_service.cc:1467] Processing CreateTablet for tablet 135da52054584a27959efdd9126cfcef (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.423573 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135da52054584a27959efdd9126cfcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.423828 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6a54ac32694640bffdf1c7f3b035cc (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.424201 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6a54ac32694640bffdf1c7f3b035cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.427116 28855 tablet_service.cc:1467] Processing CreateTablet for tablet 135da52054584a27959efdd9126cfcef (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.427589 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135da52054584a27959efdd9126cfcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.428553 28724 tablet_service.cc:1467] Processing CreateTablet for tablet 763b2e84afa64924ad5eeeeb3265a1d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.429060 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b2e84afa64924ad5eeeeb3265a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.432839 28856 tablet_service.cc:1467] Processing CreateTablet for tablet 763b2e84afa64924ad5eeeeb3265a1d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.433362 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b2e84afa64924ad5eeeeb3265a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.433614 29116 tablet_service.cc:1467] Processing CreateTablet for tablet 763b2e84afa64924ad5eeeeb3265a1d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:20.434089 29116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b2e84afa64924ad5eeeeb3265a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:20.434300 30260 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:20.436193 30260 tablet_bootstrap.cc:654] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.445381 30260 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:20.445680 30260 ts_tablet_manager.cc:1367] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 14:02:20.446771 30260 raft_consensus.cc:348] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.447093 30260 raft_consensus.cc:374] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.447227 30260 raft_consensus.cc:729] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.448150 30260 consensus_queue.cc:260] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.448880 30260 ts_tablet_manager.cc:1398] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:20.452368 30263 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:20.455142 30263 tablet_bootstrap.cc:654] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.462256 30260 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:20.464277 30260 tablet_bootstrap.cc:654] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.464592 30263 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:20.464742 30263 ts_tablet_manager.cc:1367] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 14:02:20.465682 30263 raft_consensus.cc:348] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.466020 30263 raft_consensus.cc:374] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.466171 30263 raft_consensus.cc:729] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.466536 30263 consensus_queue.cc:260] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.467470 30263 ts_tablet_manager.cc:1398] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:20.468469 30263 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:20.470886 30263 tablet_bootstrap.cc:654] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.476339 30260 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:20.476644 30260 ts_tablet_manager.cc:1367] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.005s
I20240501 14:02:20.477818 30260 raft_consensus.cc:348] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.478201 30260 raft_consensus.cc:374] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.478331 30260 raft_consensus.cc:729] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.478757 30260 consensus_queue.cc:260] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.479703 30260 ts_tablet_manager.cc:1398] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:20.480916 30260 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:20.481488 30266 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:20.482734 30260 tablet_bootstrap.cc:654] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.484162 30266 tablet_bootstrap.cc:654] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.484388 30263 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:20.484586 30263 ts_tablet_manager.cc:1367] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20240501 14:02:20.485451 30263 raft_consensus.cc:348] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.485720 30263 raft_consensus.cc:374] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.485805 30263 raft_consensus.cc:729] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.486022 30261 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:20.486068 30263 consensus_queue.cc:260] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.486739 30263 ts_tablet_manager.cc:1398] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:20.487066 30263 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:20.488221 30266 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:20.488318 30261 tablet_bootstrap.cc:654] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.488466 30266 ts_tablet_manager.cc:1367] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:02:20.488709 30263 tablet_bootstrap.cc:654] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.490317 30266 raft_consensus.cc:348] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.490651 30266 raft_consensus.cc:374] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.490809 30266 raft_consensus.cc:729] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.491135 30266 consensus_queue.cc:260] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.492506 30266 ts_tablet_manager.cc:1398] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:02:20.492900 30266 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:20.493438 30260 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:20.493621 30260 ts_tablet_manager.cc:1367] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 14:02:20.494486 30260 raft_consensus.cc:348] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.494773 30260 raft_consensus.cc:374] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.494884 30260 raft_consensus.cc:729] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.494940 30266 tablet_bootstrap.cc:654] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:20.495148 30260 consensus_queue.cc:260] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.496557 30212 raft_consensus.cc:482] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:20.496724 30212 raft_consensus.cc:504] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.497403 30263 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:20.497581 30212 leader_election.cc:288] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:20.497572 30263 ts_tablet_manager.cc:1367] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240501 14:02:20.498298 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6a54ac32694640bffdf1c7f3b035cc" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:20.498696 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6a54ac32694640bffdf1c7f3b035cc" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:20.498935 28744 raft_consensus.cc:2454] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
W20240501 14:02:20.499118 29072 leader_election.cc:341] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:20.499457 29073 leader_election.cc:302] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters: 7e7bc120bd74479ab378f779778fd1e8
I20240501 14:02:20.499948 30212 raft_consensus.cc:2781] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:20.500021 30260 ts_tablet_manager.cc:1398] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:02:20.500064 30212 raft_consensus.cc:482] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:20.500149 30212 raft_consensus.cc:3037] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.502161 30261 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:20.502094 30263 raft_consensus.cc:348] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.502298 30263 raft_consensus.cc:374] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.502336 30261 ts_tablet_manager.cc:1367] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20240501 14:02:20.502385 30263 raft_consensus.cc:729] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.502599 30263 consensus_queue.cc:260] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.503155 30212 raft_consensus.cc:504] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.503204 30261 raft_consensus.cc:348] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.503356 30263 ts_tablet_manager.cc:1398] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:02:20.503434 30261 raft_consensus.cc:374] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.503525 30261 raft_consensus.cc:729] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.503558 30266 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:20.503762 30261 consensus_queue.cc:260] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.504036 30266 ts_tablet_manager.cc:1367] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240501 14:02:20.504236 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6a54ac32694640bffdf1c7f3b035cc" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:20.504354 30261 ts_tablet_manager.cc:1398] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:20.504463 28744 raft_consensus.cc:3037] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.504557 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6a54ac32694640bffdf1c7f3b035cc" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:20.504801 29005 raft_consensus.cc:3037] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.504889 30266 raft_consensus.cc:348] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.505127 30266 raft_consensus.cc:374] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:20.505199 30266 raft_consensus.cc:729] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:20.505534 30266 consensus_queue.cc:260] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.506093 30266 ts_tablet_manager.cc:1398] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:20.507249 28744 raft_consensus.cc:2454] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:20.507432 30212 leader_election.cc:288] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:20.507582 29005 raft_consensus.cc:2454] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:20.507808 29073 leader_election.cc:302] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:20.509057 30212 raft_consensus.cc:2781] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:20.509238 30212 raft_consensus.cc:686] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:20.509529 30212 consensus_queue.cc:237] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:20.513543 28598 catalog_manager.cc:5484] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:20.516949 29988 raft_consensus.cc:482] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:20.517118 29988 raft_consensus.cc:504] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.517786 29988 leader_election.cc:288] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:20.518178 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135da52054584a27959efdd9126cfcef" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:20.518335 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135da52054584a27959efdd9126cfcef" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:20.518489 29137 raft_consensus.cc:2454] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:20.518606 28876 raft_consensus.cc:2454] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:20.518990 28678 leader_election.cc:302] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:20.519277 29988 raft_consensus.cc:2781] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:20.519405 29988 raft_consensus.cc:482] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:20.519507 29988 raft_consensus.cc:3037] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.521535 29988 raft_consensus.cc:504] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.522100 29988 leader_election.cc:288] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:20.522655 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135da52054584a27959efdd9126cfcef" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:20.522760 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135da52054584a27959efdd9126cfcef" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:20.522898 29137 raft_consensus.cc:3037] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.522994 28876 raft_consensus.cc:3037] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.525153 29137 raft_consensus.cc:2454] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:20.525215 28876 raft_consensus.cc:2454] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:20.525630 28678 leader_election.cc:302] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:20.525914 29988 raft_consensus.cc:2781] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:20.526083 29988 raft_consensus.cc:686] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Leader. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:02:20.526337 29988 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.529613 28598 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a85ba3b514663af5320cd14a9e144 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:20.563370 29988 raft_consensus.cc:482] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:20.563630 29988 raft_consensus.cc:504] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.564807 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b2e84afa64924ad5eeeeb3265a1d8" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:20.565075 29137 raft_consensus.cc:2454] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:20.565511 29988 leader_election.cc:288] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:20.565622 28678 leader_election.cc:302] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:20.565918 29988 raft_consensus.cc:2781] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:20.566031 29988 raft_consensus.cc:482] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:20.566155 29988 raft_consensus.cc:3037] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.566357 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b2e84afa64924ad5eeeeb3265a1d8" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:20.566666 28876 raft_consensus.cc:2454] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:20.568611 29988 raft_consensus.cc:504] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.569244 29988 leader_election.cc:288] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:20.569633 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b2e84afa64924ad5eeeeb3265a1d8" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:20.569689 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b2e84afa64924ad5eeeeb3265a1d8" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:20.569813 29137 raft_consensus.cc:3037] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.569916 28876 raft_consensus.cc:3037] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:20.571887 28876 raft_consensus.cc:2454] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:20.571894 29137 raft_consensus.cc:2454] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:20.572304 28678 leader_election.cc:302] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:02:20.572603 29988 raft_consensus.cc:2781] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:20.572805 29988 raft_consensus.cc:686] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Leader. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:02:20.573081 29988 consensus_queue.cc:237] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:20.576462 28598 catalog_manager.cc:5484] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a85ba3b514663af5320cd14a9e144 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:20.917346 29906 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:20.928040 30212 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:20.984408 30276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:20.984771 30276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:20.992825 30276 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:02:21.035522 29988 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:21.046947 30202 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:21.093690 30202 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:21.099586 30303 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 303.055 ms
time per row: 0.151528 ms
I20240501 14:02:21.453246 28596 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:21.462697 28725 tablet_service.cc:1467] Processing CreateTablet for tablet e821b93aa9b84ff3a968860a5b33b918 (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.462864 28985 tablet_service.cc:1467] Processing CreateTablet for tablet e821b93aa9b84ff3a968860a5b33b918 (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.463251 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e821b93aa9b84ff3a968860a5b33b918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.463364 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e821b93aa9b84ff3a968860a5b33b918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.463502 28856 tablet_service.cc:1467] Processing CreateTablet for tablet e821b93aa9b84ff3a968860a5b33b918 (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.463948 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e821b93aa9b84ff3a968860a5b33b918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.464229 28986 tablet_service.cc:1467] Processing CreateTablet for tablet ec1b3fec785f43c4a3bacedecd8472dd (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.464612 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1b3fec785f43c4a3bacedecd8472dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.468153 29118 tablet_service.cc:1467] Processing CreateTablet for tablet ec1b3fec785f43c4a3bacedecd8472dd (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.468632 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1b3fec785f43c4a3bacedecd8472dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.475252 28724 tablet_service.cc:1467] Processing CreateTablet for tablet ec1b3fec785f43c4a3bacedecd8472dd (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.475858 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1b3fec785f43c4a3bacedecd8472dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.477757 30342 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:21.479630 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8c435795a04b1b90b6e2ff914676c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.480070 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8c435795a04b1b90b6e2ff914676c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.487751 28855 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8c435795a04b1b90b6e2ff914676c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.488308 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8c435795a04b1b90b6e2ff914676c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.488457 28986 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8c435795a04b1b90b6e2ff914676c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:21.489187 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8c435795a04b1b90b6e2ff914676c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:21.496948 30342 tablet_bootstrap.cc:654] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.502292 30343 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:21.502871 30344 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:21.504212 30343 tablet_bootstrap.cc:654] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.506460 30341 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:21.506510 30342 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:21.506831 30342 ts_tablet_manager.cc:1367] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240501 14:02:21.507813 30343 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:21.507795 30342 raft_consensus.cc:348] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.508050 30343 ts_tablet_manager.cc:1367] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:02:21.508111 30342 raft_consensus.cc:374] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.508262 30342 raft_consensus.cc:729] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.508517 30341 tablet_bootstrap.cc:654] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.508651 30342 consensus_queue.cc:260] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.509044 30343 raft_consensus.cc:348] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.509348 30343 raft_consensus.cc:374] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.509377 30342 ts_tablet_manager.cc:1398] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:21.509516 30343 raft_consensus.cc:729] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.509680 30342 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:21.509912 30343 consensus_queue.cc:260] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.511440 30342 tablet_bootstrap.cc:654] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.513135 30343 ts_tablet_manager.cc:1398] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:02:21.515462 30343 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:21.516973 30344 tablet_bootstrap.cc:654] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.518460 30343 tablet_bootstrap.cc:654] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.519745 30342 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:21.520041 30342 ts_tablet_manager.cc:1367] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.008s
I20240501 14:02:21.521225 30342 raft_consensus.cc:348] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.521638 30342 raft_consensus.cc:374] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.521845 30342 raft_consensus.cc:729] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.522226 30342 consensus_queue.cc:260] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.523193 30342 ts_tablet_manager.cc:1398] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:21.525544 30343 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:21.525779 30343 ts_tablet_manager.cc:1367] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240501 14:02:21.526227 30344 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:21.526399 30344 ts_tablet_manager.cc:1367] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20240501 14:02:21.526862 30343 raft_consensus.cc:348] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.527150 30343 raft_consensus.cc:374] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.527271 30343 raft_consensus.cc:729] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.527622 30343 consensus_queue.cc:260] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.527999 30341 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:21.528278 30341 ts_tablet_manager.cc:1367] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s
I20240501 14:02:21.528386 30343 ts_tablet_manager.cc:1398] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:21.529037 30344 raft_consensus.cc:348] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.529289 30344 raft_consensus.cc:374] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.529279 30341 raft_consensus.cc:348] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.529419 30344 raft_consensus.cc:729] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.529525 30341 raft_consensus.cc:374] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.529671 30341 raft_consensus.cc:729] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.529673 30344 consensus_queue.cc:260] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.529925 30341 consensus_queue.cc:260] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.530551 30341 ts_tablet_manager.cc:1398] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:21.530858 30341 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:21.532562 30341 tablet_bootstrap.cc:654] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.533231 30344 ts_tablet_manager.cc:1398] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:02:21.533632 30344 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:21.536505 30344 tablet_bootstrap.cc:654] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.538019 30341 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:21.538283 30341 ts_tablet_manager.cc:1367] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20240501 14:02:21.539104 30341 raft_consensus.cc:348] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.539367 30341 raft_consensus.cc:374] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.539456 30341 raft_consensus.cc:729] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.539769 30344 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:21.539705 30341 consensus_queue.cc:260] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.539938 30344 ts_tablet_manager.cc:1367] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:02:21.540313 30341 ts_tablet_manager.cc:1398] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:21.540647 30341 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:21.540757 30344 raft_consensus.cc:348] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.541026 30344 raft_consensus.cc:374] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.541141 30344 raft_consensus.cc:729] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.541468 30344 consensus_queue.cc:260] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.542214 30344 ts_tablet_manager.cc:1398] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:02:21.542320 30341 tablet_bootstrap.cc:654] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:21.544595 30341 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:21.544736 30341 ts_tablet_manager.cc:1367] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:21.545507 30341 raft_consensus.cc:348] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.545658 30341 raft_consensus.cc:374] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:21.545748 30341 raft_consensus.cc:729] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:21.545984 30341 consensus_queue.cc:260] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.546583 30341 ts_tablet_manager.cc:1398] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:21.657724 30110 raft_consensus.cc:482] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:21.657949 30110 raft_consensus.cc:504] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.658568 30110 leader_election.cc:288] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:21.659183 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e821b93aa9b84ff3a968860a5b33b918" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:21.659183 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e821b93aa9b84ff3a968860a5b33b918" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:21.659415 29007 raft_consensus.cc:2454] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:21.659442 28744 raft_consensus.cc:2454] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 0.
I20240501 14:02:21.659902 28810 leader_election.cc:302] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:21.660197 30110 raft_consensus.cc:2781] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:21.660301 30110 raft_consensus.cc:482] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:21.660415 30110 raft_consensus.cc:3037] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.662370 30110 raft_consensus.cc:504] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.663045 30110 leader_election.cc:288] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Requested vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:21.663631 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e821b93aa9b84ff3a968860a5b33b918" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:21.663703 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e821b93aa9b84ff3a968860a5b33b918" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:21.663895 28744 raft_consensus.cc:3037] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.663919 29007 raft_consensus.cc:3037] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.666693 28744 raft_consensus.cc:2454] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:21.667160 28811 leader_election.cc:302] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:21.667273 29007 raft_consensus.cc:2454] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 1.
I20240501 14:02:21.667500 30336 raft_consensus.cc:2781] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:21.668123 30336 raft_consensus.cc:686] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:02:21.668502 30336 consensus_queue.cc:237] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:02:21.671633 28596 catalog_manager.cc:5484] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:02:21.691610 30038 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:21.691859 30038 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.692487 30038 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:21.693004 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:21.693127 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:21.693310 29138 raft_consensus.cc:2454] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:21.693351 28876 raft_consensus.cc:2454] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:21.693877 28940 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:21.694212 30038 raft_consensus.cc:2781] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:21.694332 30038 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:21.694456 30038 raft_consensus.cc:3037] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.696760 30038 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.697418 30038 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:21.697786 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:21.697877 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:21.697996 29138 raft_consensus.cc:3037] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.698087 28876 raft_consensus.cc:3037] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.700035 29138 raft_consensus.cc:2454] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:21.700052 28876 raft_consensus.cc:2454] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:21.700455 28940 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:21.700739 30038 raft_consensus.cc:2781] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:21.700937 30038 raft_consensus.cc:686] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:21.701205 30038 consensus_queue.cc:237] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:21.704474 28596 catalog_manager.cc:5484] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:21.764050 30038 raft_consensus.cc:482] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:21.764268 30038 raft_consensus.cc:504] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.764964 30038 leader_election.cc:288] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:21.765648 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:21.765658 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:21.765882 29138 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:21.765909 28744 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:21.766362 28940 leader_election.cc:302] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:21.766697 30038 raft_consensus.cc:2781] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:21.766822 30038 raft_consensus.cc:482] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:21.766932 30038 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.768972 30038 raft_consensus.cc:504] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.769649 30038 leader_election.cc:288] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:21.770253 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:21.770256 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:21.770483 28744 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.770483 29138 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:21.773404 29138 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:21.773399 28744 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:21.773866 28940 leader_election.cc:302] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:21.774220 30038 raft_consensus.cc:2781] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:21.774396 30038 raft_consensus.cc:686] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:21.774726 30038 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:21.777979 28596 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:02:22.094470 30336 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:22.103761 30110 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:22.120052 30038 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:22.125166 30327 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:22.245115 30358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:22.245501 30358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:22.253830 30358 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:02:22.334316 30327 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:22.341773 30327 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.804 ms
time per row: 0.119902 ms
I20240501 14:02:22.651974 28595 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:22.661914 28986 tablet_service.cc:1467] Processing CreateTablet for tablet e8b9db597e654ed48b679abf58b8a2c8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.662456 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b9db597e654ed48b679abf58b8a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.662721 29118 tablet_service.cc:1467] Processing CreateTablet for tablet efe7fed8e5d44e8a80ffbdc27619a74d (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.663076 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7c7b963588436bb778e45da469c7d5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.663298 28855 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7c7b963588436bb778e45da469c7d5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.663323 28724 tablet_service.cc:1467] Processing CreateTablet for tablet e8b9db597e654ed48b679abf58b8a2c8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.663789 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b9db597e654ed48b679abf58b8a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.663810 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7c7b963588436bb778e45da469c7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.663865 28856 tablet_service.cc:1467] Processing CreateTablet for tablet e8b9db597e654ed48b679abf58b8a2c8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.664258 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b9db597e654ed48b679abf58b8a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.663187 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe7fed8e5d44e8a80ffbdc27619a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.667311 28725 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7c7b963588436bb778e45da469c7d5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.669585 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7c7b963588436bb778e45da469c7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.669953 28723 tablet_service.cc:1467] Processing CreateTablet for tablet efe7fed8e5d44e8a80ffbdc27619a74d (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.670390 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe7fed8e5d44e8a80ffbdc27619a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.670926 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7c7b963588436bb778e45da469c7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.682086 30422 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:22.684173 30421 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:22.686250 30421 tablet_bootstrap.cc:654] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.688010 30422 tablet_bootstrap.cc:654] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.687844 28985 tablet_service.cc:1467] Processing CreateTablet for tablet efe7fed8e5d44e8a80ffbdc27619a74d (DEFAULT_TABLE table=rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:22.689174 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe7fed8e5d44e8a80ffbdc27619a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:22.701433 30421 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:22.701741 30421 ts_tablet_manager.cc:1367] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240501 14:02:22.702612 30420 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:22.702818 30421 raft_consensus.cc:348] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.703171 30421 raft_consensus.cc:374] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.703338 30421 raft_consensus.cc:729] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.703733 30421 consensus_queue.cc:260] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.705287 30421 ts_tablet_manager.cc:1398] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:02:22.705655 30421 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:22.706352 30425 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:22.707291 30422 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:22.707536 30422 ts_tablet_manager.cc:1367] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20240501 14:02:22.707408 30421 tablet_bootstrap.cc:654] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.709525 30422 raft_consensus.cc:348] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.710742 30422 raft_consensus.cc:374] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.710858 30422 raft_consensus.cc:729] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.711135 30422 consensus_queue.cc:260] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.711861 30425 tablet_bootstrap.cc:654] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.712942 30422 ts_tablet_manager.cc:1398] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:02:22.713347 30422 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:22.715323 30422 tablet_bootstrap.cc:654] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.718019 30421 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:22.718282 30421 ts_tablet_manager.cc:1367] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 14:02:22.719610 30420 tablet_bootstrap.cc:654] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.719276 30421 raft_consensus.cc:348] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.720098 30421 raft_consensus.cc:374] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.720191 30421 raft_consensus.cc:729] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.720477 30421 consensus_queue.cc:260] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.721948 30421 ts_tablet_manager.cc:1398] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:02:22.723351 30422 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:22.723590 30422 ts_tablet_manager.cc:1367] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 14:02:22.723984 30425 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:22.724164 30425 ts_tablet_manager.cc:1367] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240501 14:02:22.724949 30425 raft_consensus.cc:348] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.725106 30420 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:22.725171 30425 raft_consensus.cc:374] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.725271 30425 raft_consensus.cc:729] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.725268 30420 ts_tablet_manager.cc:1367] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20240501 14:02:22.725404 30422 raft_consensus.cc:348] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.725668 30422 raft_consensus.cc:374] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.725752 30422 raft_consensus.cc:729] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.725698 30425 consensus_queue.cc:260] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.726004 30422 consensus_queue.cc:260] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.726224 30420 raft_consensus.cc:348] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.726497 30420 raft_consensus.cc:374] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.726603 30420 raft_consensus.cc:729] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.726902 30425 ts_tablet_manager.cc:1398] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:02:22.726895 30420 consensus_queue.cc:260] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.727589 30420 ts_tablet_manager.cc:1398] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:22.727674 30422 ts_tablet_manager.cc:1398] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:22.727895 30420 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:22.731925 30425 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:22.733966 30425 tablet_bootstrap.cc:654] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.735908 30425 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:22.736073 30425 ts_tablet_manager.cc:1367] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:02:22.736717 30425 raft_consensus.cc:348] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.736907 30425 raft_consensus.cc:374] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.736987 30425 raft_consensus.cc:729] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.737251 30425 consensus_queue.cc:260] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.737882 30425 ts_tablet_manager.cc:1398] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:22.738270 30425 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:22.739850 30420 tablet_bootstrap.cc:654] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.739890 30425 tablet_bootstrap.cc:654] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:22.745322 30420 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:22.745594 30420 ts_tablet_manager.cc:1367] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20240501 14:02:22.745899 30425 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:22.746060 30425 ts_tablet_manager.cc:1367] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 14:02:22.746671 30420 raft_consensus.cc:348] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.746973 30420 raft_consensus.cc:374] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.747078 30420 raft_consensus.cc:729] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.746968 30425 raft_consensus.cc:348] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.747256 30425 raft_consensus.cc:374] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:22.747383 30425 raft_consensus.cc:729] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:22.747505 30420 consensus_queue.cc:260] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.747628 30425 consensus_queue.cc:260] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.748082 30420 ts_tablet_manager.cc:1398] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:22.750020 30425 ts_tablet_manager.cc:1398] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:02:22.801237 30327 raft_consensus.cc:482] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:22.801474 30327 raft_consensus.cc:504] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.802224 30327 leader_election.cc:288] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:22.802829 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe7fed8e5d44e8a80ffbdc27619a74d" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:22.802897 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe7fed8e5d44e8a80ffbdc27619a74d" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:22.803095 29138 raft_consensus.cc:2454] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:22.803198 28744 raft_consensus.cc:2454] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:22.803624 28940 leader_election.cc:302] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:22.803946 30327 raft_consensus.cc:2781] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:22.804071 30327 raft_consensus.cc:482] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:22.804188 30327 raft_consensus.cc:3037] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.806090 30327 raft_consensus.cc:504] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.806746 30327 leader_election.cc:288] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:22.807266 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe7fed8e5d44e8a80ffbdc27619a74d" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:22.807276 28744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe7fed8e5d44e8a80ffbdc27619a74d" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:22.807490 29138 raft_consensus.cc:3037] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.807569 28744 raft_consensus.cc:3037] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.809914 29138 raft_consensus.cc:2454] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:22.809934 28744 raft_consensus.cc:2454] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:22.810364 28942 leader_election.cc:302] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:22.810708 30327 raft_consensus.cc:2781] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:22.810916 30327 raft_consensus.cc:686] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:22.811188 30327 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.814394 28596 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:22.869823 30375 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:22.870035 30375 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.871052 30375 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:22.871320 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:22.871659 28876 raft_consensus.cc:2454] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:22.871870 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:22.872102 28743 raft_consensus.cc:2454] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 0.
I20240501 14:02:22.872123 29071 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:22.872419 30375 raft_consensus.cc:2781] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:22.872532 30375 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:22.872645 30375 raft_consensus.cc:3037] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.874724 30375 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.875396 30375 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:22.875844 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:22.875936 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:22.876072 28876 raft_consensus.cc:3037] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.876178 28743 raft_consensus.cc:3037] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.878067 28876 raft_consensus.cc:2454] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:22.878093 28743 raft_consensus.cc:2454] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 1.
I20240501 14:02:22.878527 29071 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:22.878903 30375 raft_consensus.cc:2781] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:22.879299 30375 raft_consensus.cc:686] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Leader. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:22.879617 30375 consensus_queue.cc:237] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:22.883045 28596 catalog_manager.cc:5484] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3972b170c54344a4ba61f176f837ba (127.16.36.132). New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:02:22.936195 30327 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:22.936388 30327 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.937042 30327 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:22.937584 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:02:22.937698 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:22.937834 28743 raft_consensus.cc:2454] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:22.937970 28876 raft_consensus.cc:2454] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:22.938329 28942 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:22.938661 30327 raft_consensus.cc:2781] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:22.938787 30327 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:22.938881 30327 raft_consensus.cc:3037] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.940889 30327 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.941469 30327 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:22.942062 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:22.942076 28876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:22.942322 28743 raft_consensus.cc:3037] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.942337 28876 raft_consensus.cc:3037] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:22.945334 28743 raft_consensus.cc:2454] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:22.945430 28876 raft_consensus.cc:2454] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:22.945874 28942 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:22.946233 30327 raft_consensus.cc:2781] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:22.946390 30327 raft_consensus.cc:686] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:22.946713 30327 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:02:22.949939 28596 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:02:23.257589 30327 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:23.265817 30038 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:23.314409 30434 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:23.319867 30434 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:23.368367 30437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:23.368713 30437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:02:23.372170 30038 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:23.377846 30437 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:23.384668 30450 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 269.333 ms
time per row: 0.134667 ms
I20240501 14:02:23.813228 28596 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57698:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:02:23.823387 28855 tablet_service.cc:1467] Processing CreateTablet for tablet 111ba172d7e44054872c8fc8cac8ba88 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.823972 28855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111ba172d7e44054872c8fc8cac8ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.824205 28856 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdedd39e12a49659d989f37b3287d87 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.824678 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdedd39e12a49659d989f37b3287d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.825462 28854 tablet_service.cc:1467] Processing CreateTablet for tablet 9b473e10a8ad4bd58bb44aabd9656e08 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.825842 28854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b473e10a8ad4bd58bb44aabd9656e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.825819 28985 tablet_service.cc:1467] Processing CreateTablet for tablet 111ba172d7e44054872c8fc8cac8ba88 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.826242 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111ba172d7e44054872c8fc8cac8ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.827100 28725 tablet_service.cc:1467] Processing CreateTablet for tablet 9b473e10a8ad4bd58bb44aabd9656e08 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.827553 28725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b473e10a8ad4bd58bb44aabd9656e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.829456 29118 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdedd39e12a49659d989f37b3287d87 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.829923 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdedd39e12a49659d989f37b3287d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.830997 29117 tablet_service.cc:1467] Processing CreateTablet for tablet 111ba172d7e44054872c8fc8cac8ba88 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.831074 28986 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdedd39e12a49659d989f37b3287d87 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.831363 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111ba172d7e44054872c8fc8cac8ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.831467 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdedd39e12a49659d989f37b3287d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.833927 28987 tablet_service.cc:1467] Processing CreateTablet for tablet 9b473e10a8ad4bd58bb44aabd9656e08 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:02:23.834455 28987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b473e10a8ad4bd58bb44aabd9656e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:23.841298 30503 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:23.844331 30503 tablet_bootstrap.cc:654] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.870539 30506 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:02:23.872752 30506 tablet_bootstrap.cc:654] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.882922 30505 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:23.885064 30506 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: No bootstrap required, opened a new log
I20240501 14:02:23.885361 30506 ts_tablet_manager.cc:1367] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240501 14:02:23.886721 30505 tablet_bootstrap.cc:654] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.886535 30506 raft_consensus.cc:348] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.886878 30506 raft_consensus.cc:374] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.887014 30506 raft_consensus.cc:729] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.887421 30506 consensus_queue.cc:260] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.888180 30502 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:23.888240 30506 ts_tablet_manager.cc:1398] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240501 14:02:23.888545 30503 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:23.888736 30503 ts_tablet_manager.cc:1367] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.048s user 0.030s sys 0.009s
I20240501 14:02:23.889706 30503 raft_consensus.cc:348] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.889953 30503 raft_consensus.cc:374] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.890067 30503 raft_consensus.cc:729] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.890375 30503 consensus_queue.cc:260] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.893761 30502 tablet_bootstrap.cc:654] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.894850 30503 ts_tablet_manager.cc:1398] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:02:23.896301 30503 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:02:23.896261 30502 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:23.896493 30502 ts_tablet_manager.cc:1367] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:02:23.897500 30502 raft_consensus.cc:348] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.898934 30502 raft_consensus.cc:374] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.899266 30502 raft_consensus.cc:729] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.899899 30502 consensus_queue.cc:260] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.900636 30502 ts_tablet_manager.cc:1398] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:02:23.900990 30503 tablet_bootstrap.cc:654] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.901055 30502 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:23.902863 30502 tablet_bootstrap.cc:654] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.903440 30503 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:23.903645 30503 ts_tablet_manager.cc:1367] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:02:23.904543 30503 raft_consensus.cc:348] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.904832 30505 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:23.904825 30503 raft_consensus.cc:374] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.904925 30503 raft_consensus.cc:729] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.905015 30505 ts_tablet_manager.cc:1367] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20240501 14:02:23.905210 30503 consensus_queue.cc:260] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.905803 30503 ts_tablet_manager.cc:1398] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:23.905882 30505 raft_consensus.cc:348] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.906085 30505 raft_consensus.cc:374] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.906175 30505 raft_consensus.cc:729] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.906420 30505 consensus_queue.cc:260] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.907327 30505 ts_tablet_manager.cc:1398] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:02:23.907719 30505 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:23.909652 30505 tablet_bootstrap.cc:654] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.910032 30452 raft_consensus.cc:482] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:23.910202 30452 raft_consensus.cc:504] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.910998 30452 leader_election.cc:288] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:23.911684 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b473e10a8ad4bd58bb44aabd9656e08" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:02:23.911798 30502 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:23.911895 29005 raft_consensus.cc:2454] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 0.
I20240501 14:02:23.912036 30502 ts_tablet_manager.cc:1367] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 14:02:23.912418 28679 leader_election.cc:302] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:23.912711 30452 raft_consensus.cc:2781] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:23.912834 30452 raft_consensus.cc:482] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:23.912942 30452 raft_consensus.cc:3037] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:23.912890 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b473e10a8ad4bd58bb44aabd9656e08" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:23.913400 30502 raft_consensus.cc:348] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.914196 30502 raft_consensus.cc:374] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.914494 30502 raft_consensus.cc:729] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.914674 30503 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
W20240501 14:02:23.914875 28678 leader_election.cc:341] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:02:23.914990 30502 consensus_queue.cc:260] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.916096 30452 raft_consensus.cc:504] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.916662 30503 tablet_bootstrap.cc:654] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.916710 30502 ts_tablet_manager.cc:1398] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:02:23.916785 30452 leader_election.cc:288] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:23.917055 30502 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:02:23.917428 29005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b473e10a8ad4bd58bb44aabd9656e08" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:23.917629 29005 raft_consensus.cc:3037] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:23.918761 30502 tablet_bootstrap.cc:654] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Neither blocks nor log segments found. Creating new log.
I20240501 14:02:23.920586 29005 raft_consensus.cc:2454] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:23.921068 28679 leader_election.cc:302] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:23.921223 30505 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba: No bootstrap required, opened a new log
I20240501 14:02:23.921415 30505 ts_tablet_manager.cc:1367] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240501 14:02:23.922165 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b473e10a8ad4bd58bb44aabd9656e08" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:23.922441 28875 raft_consensus.cc:3037] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:23.922257 30505 raft_consensus.cc:348] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.923034 30505 raft_consensus.cc:374] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.923115 30505 raft_consensus.cc:729] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.923400 30505 consensus_queue.cc:260] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.924822 30502 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: No bootstrap required, opened a new log
I20240501 14:02:23.924923 30505 ts_tablet_manager.cc:1398] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 14:02:23.924985 30502 ts_tablet_manager.cc:1367] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 14:02:23.925415 28875 raft_consensus.cc:2454] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343a85ba3b514663af5320cd14a9e144 in term 1.
I20240501 14:02:23.925918 30452 raft_consensus.cc:2781] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:23.925969 30502 raft_consensus.cc:348] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.926106 30452 raft_consensus.cc:686] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Leader. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:02:23.926265 30502 raft_consensus.cc:374] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.926398 30502 raft_consensus.cc:729] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.926398 30452 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:02:23.926684 30502 consensus_queue.cc:260] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.927500 30502 ts_tablet_manager.cc:1398] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 14:02:23.929517 30503 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: No bootstrap required, opened a new log
I20240501 14:02:23.929718 30503 ts_tablet_manager.cc:1367] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20240501 14:02:23.931070 30503 raft_consensus.cc:348] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.931324 30503 raft_consensus.cc:374] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:02:23.931421 30503 raft_consensus.cc:729] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:02:23.931730 30503 consensus_queue.cc:260] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.934533 30503 ts_tablet_manager.cc:1398] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 14:02:23.938838 28596 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: term changed from 0 to 1, leader changed from <none> to 343a85ba3b514663af5320cd14a9e144 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:02:23.983194 30481 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:23.983410 30481 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.984167 30481 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:23.984797 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:23.984948 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:23.985031 28875 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:23.985193 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:23.985469 28940 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:23.985760 30481 raft_consensus.cc:2781] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:23.985889 30481 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:23.985976 30481 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:23.988034 30481 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.988667 30481 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:23.989204 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:23.989490 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:23.989591 28875 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:23.989689 29138 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:23.991777 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:23.992342 28940 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:23.992551 28875 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:23.992677 30481 raft_consensus.cc:2781] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:23.992830 30481 raft_consensus.cc:686] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:23.993391 30481 consensus_queue.cc:237] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:23.996757 28596 catalog_manager.cc:5484] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:24.042534 30481 raft_consensus.cc:482] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:02:24.042824 30481 raft_consensus.cc:504] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:24.043648 30481 leader_election.cc:288] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:24.044162 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111ba172d7e44054872c8fc8cac8ba88" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:02:24.044293 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111ba172d7e44054872c8fc8cac8ba88" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:02:24.044492 28875 raft_consensus.cc:2454] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:24.044587 29138 raft_consensus.cc:2454] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 0.
I20240501 14:02:24.044952 28940 leader_election.cc:302] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:02:24.045281 30481 raft_consensus.cc:2781] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:02:24.045394 30481 raft_consensus.cc:482] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:02:24.045511 30481 raft_consensus.cc:3037] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:24.047855 30481 raft_consensus.cc:504] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:24.048458 30481 leader_election.cc:288] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:24.048944 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111ba172d7e44054872c8fc8cac8ba88" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:02:24.048965 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111ba172d7e44054872c8fc8cac8ba88" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:24.049171 29138 raft_consensus.cc:3037] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:24.049188 28875 raft_consensus.cc:3037] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:02:24.051138 29138 raft_consensus.cc:2454] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:24.051219 28875 raft_consensus.cc:2454] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 1.
I20240501 14:02:24.051582 28940 leader_election.cc:302] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:02:24.051899 30481 raft_consensus.cc:2781] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:02:24.052093 30481 raft_consensus.cc:686] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:02:24.052364 30481 consensus_queue.cc:237] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:24.055745 28596 catalog_manager.cc:5484] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:24.379668 30452 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:24.385864 30452 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:24.444038 30481 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:24.449811 30526 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:02:24.474644 30518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:24.475170 30518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:24.484026 30518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:24.500519 30038 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:02:24.506178 30038 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 270.682 ms
time per row: 0.135341 ms
I20240501 14:02:24.923184 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:40601
--local_ip_for_outbound_sockets=127.16.36.133
--tserver_master_addrs=127.16.36.190:41971
--webserver_port=34471
--webserver_interface=127.16.36.133
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:02:25.316449 30580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:25.316777 30580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:25.316919 30580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:25.324963 30580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:25.325115 30580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:25.325290 30580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 14:02:25.335781 30580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:40601
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=34471
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:25.337174 30580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:25.339368 30580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:25.355602 30588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.356158 30585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.356647 30586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:25.357403 30580 server_base.cc:1034] running on GCE node
I20240501 14:02:25.358057 30580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:25.359094 30580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:25.360317 30580 hybrid_clock.cc:648] HybridClock initialized: now 1714572145360271 us; error 36 us; skew 500 ppm
I20240501 14:02:25.360692 30580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:25.363065 30580 webserver.cc:460] Webserver started at http://127.16.36.133:34471/ using document root <none> and password file <none>
I20240501 14:02:25.363838 30580 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:25.363981 30580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:25.369786 30580 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:02:25.372820 30594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.374682 30580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:25.374883 30580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "26acba62745c48e3ba8180748d55b211"
format_stamp: "Formatted at 2024-05-01 14:02:06 on dist-test-slave-9q9t"
I20240501 14:02:25.375892 30580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:25.410966 30580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:25.412287 30580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:25.412695 30580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:25.414103 30580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:25.416354 30580 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:25.416451 30580 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.416736 30580 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:25.416841 30580 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.456498 30580 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:40601
I20240501 14:02:25.456590 30706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:40601 every 8 connection(s)
I20240501 14:02:25.458334 30580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:02:25.462787 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 30580
I20240501 14:02:25.463680 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.134:44241
--local_ip_for_outbound_sockets=127.16.36.134
--tserver_master_addrs=127.16.36.190:41971
--webserver_port=34609
--webserver_interface=127.16.36.134
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:25.474323 30707 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:25.474715 30707 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:25.475612 30707 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:25.477329 28604 ts_manager.cc:194] Re-registered known tserver with Master: 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:02:25.478544 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:42489
W20240501 14:02:25.860193 30711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:25.860520 30711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:25.860620 30711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:25.868708 30711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:25.868847 30711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:25.868969 30711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.134
I20240501 14:02:25.878840 30711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.134:44241
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.134
--webserver_port=34609
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.134
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:25.880486 30711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:25.882432 30711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:25.895763 30721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.896354 30718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:25.896929 30719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:25.897619 30711 server_base.cc:1034] running on GCE node
I20240501 14:02:25.898523 30711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:25.899749 30711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:25.901285 30711 hybrid_clock.cc:648] HybridClock initialized: now 1714572145901115 us; error 82 us; skew 500 ppm
I20240501 14:02:25.901785 30711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:25.911206 30711 webserver.cc:460] Webserver started at http://127.16.36.134:34609/ using document root <none> and password file <none>
I20240501 14:02:25.912061 30711 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:25.912204 30711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:25.918287 30711 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:02:25.921658 30727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.923192 30711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:02:25.923408 30711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8fc5d8222c164a679e7a99966da1eaaf"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:25.924247 30711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:25.945047 30711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:25.945916 30711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:25.946249 30711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:25.947459 30711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:25.949395 30711 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:25.949481 30711 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:25.949617 30711 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:25.949676 30711 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.001214 30711 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.134:44241
I20240501 14:02:26.001235 30840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.134:44241 every 8 connection(s)
I20240501 14:02:26.003043 30711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:02:26.005880 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 30711
I20240501 14:02:26.006964 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.135:42127
--local_ip_for_outbound_sockets=127.16.36.135
--tserver_master_addrs=127.16.36.190:41971
--webserver_port=44841
--webserver_interface=127.16.36.135
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:26.028877 30841 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:26.029304 30841 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:26.030200 30841 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:26.031924 28604 ts_manager.cc:194] Re-registered known tserver with Master: 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:02:26.033085 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.134:35879
W20240501 14:02:26.404686 30846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:26.405068 30846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:26.405230 30846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:26.413525 30846 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:26.413671 30846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:26.413872 30846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.135
I20240501 14:02:26.423756 30846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.135:42127
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.135
--webserver_port=44841
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.135
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:26.425174 30846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:26.427001 30846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:26.439786 30852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:26.440375 30851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:26.440990 30854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:26.441747 30846 server_base.cc:1034] running on GCE node
I20240501 14:02:26.442376 30846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:26.443616 30846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:26.444847 30846 hybrid_clock.cc:648] HybridClock initialized: now 1714572146444788 us; error 54 us; skew 500 ppm
I20240501 14:02:26.445159 30846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:26.447551 30846 webserver.cc:460] Webserver started at http://127.16.36.135:44841/ using document root <none> and password file <none>
I20240501 14:02:26.448259 30846 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:26.448369 30846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:26.454308 30846 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 14:02:26.457513 30860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.459059 30846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:02:26.459224 30846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a0e901103604414c829b40edf9f24c28"
format_stamp: "Formatted at 2024-05-01 14:02:07 on dist-test-slave-9q9t"
I20240501 14:02:26.460151 30846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:26.481982 30707 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
I20240501 14:02:26.487521 30846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:26.488380 30846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:26.488684 30846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:26.489638 30846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:26.491336 30846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:26.491421 30846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.491559 30846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:26.491613 30846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.524783 30846 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.135:42127
I20240501 14:02:26.524850 30972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.135:42127 every 8 connection(s)
I20240501 14:02:26.526314 30846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:02:26.536365 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 30846
I20240501 14:02:26.537138 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.136:36399
--local_ip_for_outbound_sockets=127.16.36.136
--tserver_master_addrs=127.16.36.190:41971
--webserver_port=43511
--webserver_interface=127.16.36.136
--builtin_ntp_servers=127.16.36.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:02:26.540704 30973 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:26.541064 30973 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:26.542068 30973 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:26.543810 28604 ts_manager.cc:194] Re-registered known tserver with Master: a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:02:26.545018 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.135:36541
W20240501 14:02:26.923249 30978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:26.923596 30978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:02:26.923714 30978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:26.932538 30978 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:02:26.932679 30978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:02:26.932857 30978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.136
I20240501 14:02:26.942495 30978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.136:36399
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.136
--webserver_port=43511
--tserver_master_addrs=127.16.36.190:41971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.136
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:02:26.943989 30978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:02:26.945865 30978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:02:26.959322 30984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:26.959869 30985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:02:26.960515 30987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:02:26.961234 30978 server_base.cc:1034] running on GCE node
I20240501 14:02:26.961839 30978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:02:26.962795 30978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:02:26.963999 30978 hybrid_clock.cc:648] HybridClock initialized: now 1714572146963937 us; error 53 us; skew 500 ppm
I20240501 14:02:26.964323 30978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:02:26.966857 30978 webserver.cc:460] Webserver started at http://127.16.36.136:43511/ using document root <none> and password file <none>
I20240501 14:02:26.967895 30978 fs_manager.cc:359] Metadata directory not provided
I20240501 14:02:26.968030 30978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:02:26.973533 30978 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:02:26.976655 30993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:26.978055 30978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:26.978222 30978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "826ccf8358b6453d88bcb132290b82a5"
format_stamp: "Formatted at 2024-05-01 14:02:08 on dist-test-slave-9q9t"
I20240501 14:02:26.979123 30978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:02:27.003203 30978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:02:27.004051 30978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:02:27.004354 30978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:02:27.005419 30978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:02:27.007095 30978 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:02:27.007180 30978 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:27.007323 30978 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:02:27.007380 30978 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:02:27.036455 30841 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
I20240501 14:02:27.047825 30978 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.136:36399
I20240501 14:02:27.047892 31105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.136:36399 every 8 connection(s)
I20240501 14:02:27.049580 30978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:02:27.059589 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 30978
I20240501 14:02:27.066387 31106 heartbeater.cc:347] Connected to a master server at 127.16.36.190:41971
I20240501 14:02:27.066776 31106 heartbeater.cc:464] Registering TS with master...
I20240501 14:02:27.067698 31106 heartbeater.cc:511] Master 127.16.36.190:41971 requested a full tablet report, sending...
I20240501 14:02:27.069435 28604 ts_manager.cc:194] Re-registered known tserver with Master: 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
I20240501 14:02:27.070595 28604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.136:45545
W20240501 14:02:27.255054 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 61921us
I20240501 14:02:27.549834 30973 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
I20240501 14:02:28.073402 31106 heartbeater.cc:503] Master 127.16.36.190:41971 was elected leader, sending a full tablet report...
I20240501 14:02:28.348759 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:28.364331 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.385059 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:28.389261 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.503991 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.514820 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:28.515805 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:28.544446 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:28.547858 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.559504 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:28.570117 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.581776 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.583935 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.603644 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:28.647763 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:28.653113 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:29.387787 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:29.387802 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.412554 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:02:29.483915 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 53917.5us
I20240501 14:02:29.500653 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.509913 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:29.528316 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.528506 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:29.529273 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.543059 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.556893 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:29.568892 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:29.612625 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:29.649708 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:29.664161 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.668222 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:29.684995 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:29.686780 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.700006 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.726845 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:29.763728 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:29.768030 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.796877 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.801195 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:29.841125 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:02:29.907241 28940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 43506.9us
I20240501 14:02:30.205055 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:30.213573 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:30.253906 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:30.282366 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:30.285962 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:30.305676 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:30.331635 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:30.393620 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:30.430038 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:30.439675 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:30.446748 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:30.447187 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:30.455994 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:30.459564 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:30.487082 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:30.489959 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.176424 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.209300 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:31.253563 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.256199 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.280047 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:31.283308 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:31.325762 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:31.340155 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:31.359948 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:31.383072 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:31.401041 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.423271 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:31.434191 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.447069 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.447455 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.480650 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:31.522529 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:31.550189 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.568818 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:31.599622 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.600883 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.611230 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:31.632848 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:31.692958 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.081893 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:02:32.087690 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 138929us
W20240501 14:02:32.094769 29071 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:37325, user_credentials={real_user=slave}} blocked reactor thread for 149144us
I20240501 14:02:32.152175 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:32.152665 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:32.173807 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.229470 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.235862 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:32.263970 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:32.322377 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.388962 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.433135 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:32.434892 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:32.435307 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:32.459122 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.474241 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.504514 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.581454 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:32.913645 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.956801 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:32.960307 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:32.981855 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:32.990293 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:33.052291 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.073485 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:33.102865 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.121721 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:33.130914 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:33.134768 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:33.188364 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.204715 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.249920 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:33.250414 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.262918 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.366757 29137 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:33.382905 28876 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:33.384608 31880 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:33.424621 28744 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:33.427436 31884 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:33.447577 31880 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:33.457248 28876 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:33.471264 28595 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:33.478833 28743 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
W20240501 14:02:33.525674 29073 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60b3db79ff754e8bbe9e5865f7104312. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:33.589075 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.636062 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:33.639205 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:33.655542 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.696976 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:33.699916 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.711284 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:33.722538 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:33.923892 29137 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:33.933460 29007 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:33.933460 28876 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:33.934767 31732 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 14:02:33.941300 31880 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:33.955518 29072 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2020fe37059a4e74bb45190398e4e177. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:34.008853 32009 ts_tablet_manager.cc:902] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:34.027175 31880 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:34.031491 29006 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:34.046877 28596 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:34.051079 28876 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:34.069036 32009 tablet_copy_client.cc:308] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:34.123984 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 60b3db79ff754e8bbe9e5865f7104312 from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:53567)
I20240501 14:02:34.124320 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 60b3db79ff754e8bbe9e5865f7104312 from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:53567: session id = a0e901103604414c829b40edf9f24c28-60b3db79ff754e8bbe9e5865f7104312
I20240501 14:02:34.133117 29158 tablet_copy_source_session.cc:205] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:34.145573 32009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b3db79ff754e8bbe9e5865f7104312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:34.208918 29137 consensus_queue.cc:237] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:34.268690 29006 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:34.269528 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86bfedcdabe44e7ab135674163935786. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:34.269637 32018 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:34.273505 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:34.276948 28743 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:34.277639 32018 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:34.314394 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:34.317660 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:02:34.326107 32062 log.cc:924] Time spent T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240501 14:02:34.333911 32018 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.342243 29006 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.349452 28604 catalog_manager.cc:5484] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.349725 28743 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.366554 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:34.367178 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:34.406162 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:34.393100 32009 tablet_copy_client.cc:783] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:34.411293 32009 tablet_copy_client.cc:647] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:34.415678 32009 tablet_copy_client.cc:523] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:34.424019 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:34.439570 32080 ts_tablet_manager.cc:902] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:34.449098 32009 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:34.460476 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:34.503793 32009 log.cc:824] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:34.581168 32080 tablet_copy_client.cc:308] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:34.612782 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 2020fe37059a4e74bb45190398e4e177 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:55989)
I20240501 14:02:34.613055 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 2020fe37059a4e74bb45190398e4e177 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:55989: session id = 8fc5d8222c164a679e7a99966da1eaaf-2020fe37059a4e74bb45190398e4e177
I20240501 14:02:34.616987 29158 tablet_copy_source_session.cc:205] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:34.622968 32080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2020fe37059a4e74bb45190398e4e177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:34.643849 32080 tablet_copy_client.cc:783] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:34.644544 32080 tablet_copy_client.cc:647] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:34.647548 32080 tablet_copy_client.cc:523] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:34.652475 32080 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:34.684360 32009 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:34.685015 32009 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:34.685622 32009 ts_tablet_manager.cc:1367] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.237s user 0.077s sys 0.008s
I20240501 14:02:34.684441 32080 log.cc:824] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:34.693060 32009 raft_consensus.cc:348] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:34.693684 32009 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:34.694489 32009 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:34.701517 32009 ts_tablet_manager.cc:1398] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20240501 14:02:34.708636 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-60b3db79ff754e8bbe9e5865f7104312 received from {username='slave'} at 127.16.36.135:53567
I20240501 14:02:34.708835 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session a0e901103604414c829b40edf9f24c28-60b3db79ff754e8bbe9e5865f7104312 on tablet 60b3db79ff754e8bbe9e5865f7104312 with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:34.712656 32009 ts_tablet_manager.cc:902] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:34.772917 32009 tablet_copy_client.cc:308] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:34.774094 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 86bfedcdabe44e7ab135674163935786 from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:53567)
I20240501 14:02:34.774307 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 86bfedcdabe44e7ab135674163935786 from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:53567: session id = a0e901103604414c829b40edf9f24c28-86bfedcdabe44e7ab135674163935786
I20240501 14:02:34.777557 29158 tablet_copy_source_session.cc:205] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:34.779461 32009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bfedcdabe44e7ab135674163935786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:34.811991 32009 tablet_copy_client.cc:783] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:34.812330 32009 tablet_copy_client.cc:647] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:34.818889 32009 tablet_copy_client.cc:523] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:34.819944 29137 consensus_queue.cc:237] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:34.822243 32080 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:34.822984 32080 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:34.823531 32080 ts_tablet_manager.cc:1367] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.171s user 0.070s sys 0.005s
I20240501 14:02:34.825546 32009 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:34.830403 32080 raft_consensus.cc:348] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:34.831012 32080 raft_consensus.cc:729] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:34.831894 32080 consensus_queue.cc:260] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:02:34.832762 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f516f01e242d4b8789130201f748f3ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:34.834096 29006 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:34.834861 32018 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:34.836870 28876 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:34.837803 32018 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:34.851613 32080 ts_tablet_manager.cc:1398] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.028s user 0.009s sys 0.007s
I20240501 14:02:34.854431 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-2020fe37059a4e74bb45190398e4e177 received from {username='slave'} at 127.16.36.134:55989
I20240501 14:02:34.854614 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-2020fe37059a4e74bb45190398e4e177 on tablet 2020fe37059a4e74bb45190398e4e177 with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:34.861601 32018 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.867828 29006 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.867995 28876 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.874264 28599 catalog_manager.cc:5484] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:34.917147 30928 raft_consensus.cc:1203] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:34.966300 32009 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:34.967001 32009 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:34.967391 32009 ts_tablet_manager.cc:1367] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.142s user 0.057s sys 0.011s
I20240501 14:02:34.970618 32009 raft_consensus.cc:348] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:34.970938 32009 raft_consensus.cc:729] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:34.977999 32009 consensus_queue.cc:260] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:34.979470 32009 ts_tablet_manager.cc:1398] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240501 14:02:34.980772 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-86bfedcdabe44e7ab135674163935786 received from {username='slave'} at 127.16.36.135:53567
I20240501 14:02:34.980980 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session a0e901103604414c829b40edf9f24c28-86bfedcdabe44e7ab135674163935786 on tablet 86bfedcdabe44e7ab135674163935786 with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:35.048609 30795 raft_consensus.cc:1203] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:35.201419 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.231647 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:35.236688 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:35.263944 32009 ts_tablet_manager.cc:902] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:35.295729 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:35.309677 29007 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:35.317677 29137 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:35.318287 30928 raft_consensus.cc:1203] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:35.320621 31934 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:35.327105 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.328615 28743 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:35.329278 31934 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:35.342939 31837 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:35.343864 28743 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:35.349653 29137 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:35.358342 28595 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:35.412770 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:35.429867 32009 tablet_copy_client.cc:308] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:35.431711 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet f516f01e242d4b8789130201f748f3ea from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:53567)
I20240501 14:02:35.431967 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet f516f01e242d4b8789130201f748f3ea from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:53567: session id = a0e901103604414c829b40edf9f24c28-f516f01e242d4b8789130201f748f3ea
I20240501 14:02:35.435714 29158 tablet_copy_source_session.cc:205] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:35.437831 32009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f516f01e242d4b8789130201f748f3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:35.439224 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.442884 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.476631 32009 tablet_copy_client.cc:783] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:35.476985 32009 tablet_copy_client.cc:647] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:35.482751 32009 tablet_copy_client.cc:523] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:02:35.494935 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efe7fed8e5d44e8a80ffbdc27619a74d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:35.500094 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:35.507403 32009 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:35.521564 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.580787 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:02:35.580852 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 65182.3us
I20240501 14:02:35.581396 29137 consensus_queue.cc:237] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:35.587970 28876 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:35.587961 29007 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:35.588840 31732 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:35.599187 31732 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:35.605046 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:35.607887 32018 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:35.608845 29007 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:35.614538 28598 catalog_manager.cc:5484] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:35.617545 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.620293 28876 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:35.665974 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:02:35.683557 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f516f01e242d4b8789130201f748f3ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:35.684417 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.684607 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:35.684805 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:35.827323 32009 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:35.827844 32009 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:35.828195 32009 ts_tablet_manager.cc:1367] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.321s user 0.061s sys 0.007s
I20240501 14:02:35.829454 32009 raft_consensus.cc:348] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:35.830210 32009 raft_consensus.cc:729] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:35.830577 32009 consensus_queue.cc:260] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:35.839800 32009 ts_tablet_manager.cc:1398] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240501 14:02:35.841099 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-f516f01e242d4b8789130201f748f3ea received from {username='slave'} at 127.16.36.135:53567
I20240501 14:02:35.841270 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session a0e901103604414c829b40edf9f24c28-f516f01e242d4b8789130201f748f3ea on tablet f516f01e242d4b8789130201f748f3ea with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:35.883081 32316 ts_tablet_manager.cc:902] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:35.926267 28743 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:35.944643 29007 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:35.947925 32064 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:35.969805 28876 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:35.971066 32064 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:35.976066 30928 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:36.035717 32330 ts_tablet_manager.cc:902] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:36.038101 29007 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
W20240501 14:02:36.047919 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b473e10a8ad4bd58bb44aabd9656e08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.031937 32333 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.057729 28598 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.066340 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.076284 28876 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:36.076896 32316 tablet_copy_client.cc:308] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:36.095468 32330 tablet_copy_client.cc:308] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:36.166549 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:36.189083 28875 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.208508 29138 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.210856 32174 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.213825 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:36.246145 28743 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.247495 32174 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.249330 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet f516f01e242d4b8789130201f748f3ea from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:48133)
I20240501 14:02:36.249542 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet f516f01e242d4b8789130201f748f3ea from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:48133: session id = 26acba62745c48e3ba8180748d55b211-f516f01e242d4b8789130201f748f3ea
I20240501 14:02:36.253742 29158 tablet_copy_source_session.cc:205] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:36.259609 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.260304 32330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f516f01e242d4b8789130201f748f3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:36.290223 32330 tablet_copy_client.cc:783] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:36.291736 32330 tablet_copy_client.cc:647] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:36.296337 32330 tablet_copy_client.cc:523] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:36.301342 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.301499 32330 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:36.303194 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:36.316777 32330 log.cc:824] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: Log is configured to *not* fsync() on all Append() calls
W20240501 14:02:36.330283 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 99130.8us
I20240501 14:02:36.341930 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:36.363266 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:36.366309 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet efe7fed8e5d44e8a80ffbdc27619a74d from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52677)
I20240501 14:02:36.366616 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet efe7fed8e5d44e8a80ffbdc27619a74d from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52677: session id = 8fc5d8222c164a679e7a99966da1eaaf-efe7fed8e5d44e8a80ffbdc27619a74d
I20240501 14:02:36.371074 29027 tablet_copy_source_session.cc:205] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:36.378652 32383 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
W20240501 14:02:36.384110 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6feda9b53e484e7ea40a5890b7d51ac4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.388558 32330 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:36.389259 32330 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:36.389823 32330 ts_tablet_manager.cc:1367] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.089s user 0.076s sys 0.004s
I20240501 14:02:36.396872 32316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe7fed8e5d44e8a80ffbdc27619a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:36.397257 28743 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.400163 28603 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.403738 29138 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.407778 32316 tablet_copy_client.cc:783] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:36.408145 32316 tablet_copy_client.cc:647] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:36.411373 32316 tablet_copy_client.cc:523] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:36.413733 32330 raft_consensus.cc:348] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:36.415274 32330 raft_consensus.cc:729] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:36.416285 32316 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:36.416884 32330 consensus_queue.cc:260] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:36.430042 32330 ts_tablet_manager.cc:1398] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.040s user 0.013s sys 0.000s
I20240501 14:02:36.431640 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-f516f01e242d4b8789130201f748f3ea received from {username='slave'} at 127.16.36.133:48133
I20240501 14:02:36.431833 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 26acba62745c48e3ba8180748d55b211-f516f01e242d4b8789130201f748f3ea on tablet f516f01e242d4b8789130201f748f3ea with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:36.490183 29007 consensus_queue.cc:237] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:36.518800 32403 ts_tablet_manager.cc:902] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:36.559669 28875 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.565969 32255 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.588721 28743 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.589751 32255 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.616674 30662 raft_consensus.cc:1203] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:36.640093 32371 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.644721 28875 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.651336 28743 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.654141 28603 catalog_manager.cc:5484] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.668866 32403 tablet_copy_client.cc:308] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
W20240501 14:02:36.703696 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb04ec77da194832aa63d69aba307de3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.714097 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 9b473e10a8ad4bd58bb44aabd9656e08 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43925)
I20240501 14:02:36.714453 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 9b473e10a8ad4bd58bb44aabd9656e08 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43925: session id = 826ccf8358b6453d88bcb132290b82a5-9b473e10a8ad4bd58bb44aabd9656e08
I20240501 14:02:36.727768 28765 tablet_copy_source_session.cc:205] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:36.728441 28875 consensus_queue.cc:237] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:36.736966 32403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b473e10a8ad4bd58bb44aabd9656e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:36.766834 28743 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.768056 32386 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.773991 29007 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.775592 32384 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.783026 32403 tablet_copy_client.cc:783] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:36.783808 32403 tablet_copy_client.cc:647] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:36.787542 32403 tablet_copy_client.cc:523] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:02:36.789644 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e821b93aa9b84ff3a968860a5b33b918. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.790736 32316 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:36.791241 32316 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:36.791555 32316 ts_tablet_manager.cc:1367] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.375s user 0.056s sys 0.004s
I20240501 14:02:36.792618 32316 raft_consensus.cc:348] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:36.792897 32316 raft_consensus.cc:729] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:36.793206 32316 consensus_queue.cc:260] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:36.799721 32403 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:36.799557 32386 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:36.804076 29006 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:36.808254 28743 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:36.811244 28598 catalog_manager.cc:5484] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:36.830857 32316 ts_tablet_manager.cc:1398] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.039s user 0.005s sys 0.010s
I20240501 14:02:36.835444 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-efe7fed8e5d44e8a80ffbdc27619a74d received from {username='slave'} at 127.16.36.134:52677
I20240501 14:02:36.835659 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-efe7fed8e5d44e8a80ffbdc27619a74d on tablet efe7fed8e5d44e8a80ffbdc27619a74d with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:36.839124 32457 ts_tablet_manager.cc:902] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:36.846331 30795 raft_consensus.cc:1203] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:36.860345 32403 log.cc:824] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:02:36.868711 29006 consensus_queue.cc:237] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
W20240501 14:02:36.873682 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 111ba172d7e44054872c8fc8cac8ba88. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:36.875293 28875 raft_consensus.cc:1261] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.876760 32371 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.877624 29138 raft_consensus.cc:1261] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:36.879252 32371 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:36.898236 32457 tablet_copy_client.cc:308] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:36.912237 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:60541)
I20240501 14:02:36.912609 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:60541: session id = 826ccf8358b6453d88bcb132290b82a5-6feda9b53e484e7ea40a5890b7d51ac4
I20240501 14:02:36.919412 28896 tablet_copy_source_session.cc:205] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:36.920646 32371 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.925603 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6feda9b53e484e7ea40a5890b7d51ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:36.925511 28873 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.933022 28595 catalog_manager.cc:5484] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.940315 29138 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:36.944202 32457 tablet_copy_client.cc:783] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:36.944602 32457 tablet_copy_client.cc:647] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:36.953830 32457 tablet_copy_client.cc:523] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:36.958012 32457 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:37.029536 32403 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.030480 32403 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:37.031071 32403 ts_tablet_manager.cc:1367] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.236s user 0.077s sys 0.029s
I20240501 14:02:37.043197 32403 raft_consensus.cc:348] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.049778 32403 raft_consensus.cc:729] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:37.053215 32403 consensus_queue.cc:260] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.069190 32503 ts_tablet_manager.cc:902] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:37.069484 32403 ts_tablet_manager.cc:1398] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.038s user 0.009s sys 0.004s
I20240501 14:02:37.077780 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-9b473e10a8ad4bd58bb44aabd9656e08 received from {username='slave'} at 127.16.36.136:43925
I20240501 14:02:37.077989 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-9b473e10a8ad4bd58bb44aabd9656e08 on tablet 9b473e10a8ad4bd58bb44aabd9656e08 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:37.089927 29138 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:37.093807 32503 tablet_copy_client.cc:308] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:37.103135 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet fb04ec77da194832aa63d69aba307de3 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43429)
I20240501 14:02:37.103730 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet fb04ec77da194832aa63d69aba307de3 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43429: session id = 826ccf8358b6453d88bcb132290b82a5-fb04ec77da194832aa63d69aba307de3
I20240501 14:02:37.107908 28743 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:37.109570 29027 tablet_copy_source_session.cc:205] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.111380 32503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb04ec77da194832aa63d69aba307de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.119269 29137 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.120059 28873 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.121312 32457 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.121361 28744 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.121860 32457 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:37.122196 32457 ts_tablet_manager.cc:1367] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.164s user 0.067s sys 0.009s
I20240501 14:02:37.122164 32278 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:37.123502 32457 raft_consensus.cc:348] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.123780 32457 raft_consensus.cc:729] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
W20240501 14:02:37.123836 29073 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de161b10c84f4bcc85f8cf7b199f99db. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:37.123809 32333 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:37.124081 32457 consensus_queue.cc:260] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.126256 32457 ts_tablet_manager.cc:1398] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:02:37.127446 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-6feda9b53e484e7ea40a5890b7d51ac4 received from {username='slave'} at 127.16.36.136:60541
I20240501 14:02:37.127758 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-6feda9b53e484e7ea40a5890b7d51ac4 on tablet 6feda9b53e484e7ea40a5890b7d51ac4 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:37.130842 32278 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:37.139873 32503 tablet_copy_client.cc:783] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.142706 32503 tablet_copy_client.cc:647] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.153659 32503 tablet_copy_client.cc:523] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.159076 29006 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.161011 32333 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:37.162586 28680 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0bf08cbb6d347d783b1fa7762c802df. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:37.172497 28744 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.174145 32550 ts_tablet_manager.cc:902] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:37.180631 28598 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.170548 32278 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.197607 28873 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.201174 32550 tablet_copy_client.cc:308] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:37.209364 32503 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:37.215994 29138 consensus_queue.cc:237] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:37.219064 31061 raft_consensus.cc:1203] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.222769 32333 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.227819 29006 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.256208 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet e821b93aa9b84ff3a968860a5b33b918 from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:49903)
I20240501 14:02:37.256506 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet e821b93aa9b84ff3a968860a5b33b918 from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:49903: session id = 26acba62745c48e3ba8180748d55b211-e821b93aa9b84ff3a968860a5b33b918
I20240501 14:02:37.260927 28896 tablet_copy_source_session.cc:205] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.262702 32550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e821b93aa9b84ff3a968860a5b33b918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.263087 28598 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.271102 32550 tablet_copy_client.cc:783] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.271531 32550 tablet_copy_client.cc:647] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
W20240501 14:02:37.272243 29072 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39f1ab3a94db41b08ac90bba52418a93. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:37.274717 29006 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.275457 32558 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:37.278538 29137 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:37.283882 28743 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.286036 32558 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:37.291208 32550 tablet_copy_client.cc:523] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.304836 32550 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:37.313234 32558 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:37.315412 29007 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:37.328060 28743 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:37.328303 28598 catalog_manager.cc:5484] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:37.330715 32503 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.331234 32503 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:37.331575 32503 ts_tablet_manager.cc:1367] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.122s user 0.060s sys 0.009s
I20240501 14:02:37.332536 32503 raft_consensus.cc:348] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.332814 32503 raft_consensus.cc:729] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:37.338052 32503 consensus_queue.cc:260] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.339170 32503 ts_tablet_manager.cc:1398] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:02:37.342700 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-fb04ec77da194832aa63d69aba307de3 received from {username='slave'} at 127.16.36.136:43429
I20240501 14:02:37.342844 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-fb04ec77da194832aa63d69aba307de3 on tablet fb04ec77da194832aa63d69aba307de3 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:37.478319 32550 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.480275 32550 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:37.481448 32550 ts_tablet_manager.cc:1367] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.177s user 0.056s sys 0.009s
I20240501 14:02:37.482901 32503 ts_tablet_manager.cc:902] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:37.484743 32550 raft_consensus.cc:348] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:37.485674 32550 raft_consensus.cc:729] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:37.486284 32550 consensus_queue.cc:260] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:37.512897 29007 consensus_queue.cc:237] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:37.518137 32503 tablet_copy_client.cc:308] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:37.519558 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 111ba172d7e44054872c8fc8cac8ba88 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43429)
I20240501 14:02:37.519801 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 111ba172d7e44054872c8fc8cac8ba88 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43429: session id = 826ccf8358b6453d88bcb132290b82a5-111ba172d7e44054872c8fc8cac8ba88
I20240501 14:02:37.523319 31061 raft_consensus.cc:1203] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.527067 32627 ts_tablet_manager.cc:902] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:37.531569 29027 tablet_copy_source_session.cc:205] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.533318 32503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111ba172d7e44054872c8fc8cac8ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.552359 32627 tablet_copy_client.cc:308] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:37.553347 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet de161b10c84f4bcc85f8cf7b199f99db from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:53567)
I20240501 14:02:37.553560 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet de161b10c84f4bcc85f8cf7b199f99db from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:53567: session id = a0e901103604414c829b40edf9f24c28-de161b10c84f4bcc85f8cf7b199f99db
I20240501 14:02:37.556514 29158 tablet_copy_source_session.cc:205] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.560216 32627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de161b10c84f4bcc85f8cf7b199f99db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.564587 32629 ts_tablet_manager.cc:902] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:37.568514 28873 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.569265 32371 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:37.577489 28743 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:37.578222 32503 tablet_copy_client.cc:783] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.578284 32371 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:37.578513 32503 tablet_copy_client.cc:647] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.579216 32627 tablet_copy_client.cc:783] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.583494 32627 tablet_copy_client.cc:647] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.584004 32503 tablet_copy_client.cc:523] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.586025 32627 tablet_copy_client.cc:523] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.586164 32550 ts_tablet_manager.cc:1398] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.104s user 0.002s sys 0.009s
I20240501 14:02:37.598202 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-e821b93aa9b84ff3a968860a5b33b918 received from {username='slave'} at 127.16.36.133:49903
I20240501 14:02:37.598377 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 26acba62745c48e3ba8180748d55b211-e821b93aa9b84ff3a968860a5b33b918 on tablet e821b93aa9b84ff3a968860a5b33b918 with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:37.601074 31061 raft_consensus.cc:1203] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.602105 32503 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:37.607272 32627 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:37.614941 32371 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:37.616118 28743 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:37.623386 28873 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:37.626051 28603 catalog_manager.cc:5484] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
W20240501 14:02:37.633443 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12510f71b6a742ab8cc359243de8119b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:37.666209 32629 tablet_copy_client.cc:308] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:37.714243 32627 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.714823 32627 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:37.721565 32627 ts_tablet_manager.cc:1367] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.114s user 0.056s sys 0.011s
I20240501 14:02:37.722525 32627 raft_consensus.cc:348] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:37.722994 32627 raft_consensus.cc:729] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:37.724148 32627 consensus_queue.cc:260] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:37.727877 30662 raft_consensus.cc:1203] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.735767 32627 ts_tablet_manager.cc:1398] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.014s user 0.002s sys 0.003s
I20240501 14:02:37.736793 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-de161b10c84f4bcc85f8cf7b199f99db received from {username='slave'} at 127.16.36.135:53567
I20240501 14:02:37.736943 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session a0e901103604414c829b40edf9f24c28-de161b10c84f4bcc85f8cf7b199f99db on tablet de161b10c84f4bcc85f8cf7b199f99db with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:37.746511 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet e0bf08cbb6d347d783b1fa7762c802df from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34125)
I20240501 14:02:37.746780 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet e0bf08cbb6d347d783b1fa7762c802df from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34125: session id = a0e901103604414c829b40edf9f24c28-e0bf08cbb6d347d783b1fa7762c802df
I20240501 14:02:37.749838 28765 tablet_copy_source_session.cc:205] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.758272 32503 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:37.758805 32503 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:37.759083 32503 ts_tablet_manager.cc:1367] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.157s user 0.058s sys 0.008s
I20240501 14:02:37.759902 32503 raft_consensus.cc:348] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.760090 32629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bf08cbb6d347d783b1fa7762c802df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.760144 32503 raft_consensus.cc:729] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:37.760423 32503 consensus_queue.cc:260] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:37.762468 32503 ts_tablet_manager.cc:1398] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:02:37.763387 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-111ba172d7e44054872c8fc8cac8ba88 received from {username='slave'} at 127.16.36.136:43429
I20240501 14:02:37.763590 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-111ba172d7e44054872c8fc8cac8ba88 on tablet 111ba172d7e44054872c8fc8cac8ba88 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:37.778882 32667 ts_tablet_manager.cc:902] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:37.812472 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:37.853014 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:37.856371 31061 raft_consensus.cc:1203] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:37.868072 32629 tablet_copy_client.cc:783] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:37.868492 32629 tablet_copy_client.cc:647] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:37.871922 29007 consensus_queue.cc:237] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:37.872982 32667 tablet_copy_client.cc:308] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:37.873891 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 39f1ab3a94db41b08ac90bba52418a93 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:55989)
I20240501 14:02:37.874106 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 39f1ab3a94db41b08ac90bba52418a93 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:55989: session id = 8fc5d8222c164a679e7a99966da1eaaf-39f1ab3a94db41b08ac90bba52418a93
I20240501 14:02:37.877661 29158 tablet_copy_source_session.cc:205] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:37.892155 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1ab3a94db41b08ac90bba52418a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:37.892728 31061 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.899896 28744 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.906421 32371 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.910390 32255 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240501 14:02:37.912456 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 85795.8us
I20240501 14:02:37.921849 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:37.924851 28873 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:37.926085 32629 tablet_copy_client.cc:523] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:37.930473 32696 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:37.939327 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:37.949117 32629 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:37.965631 28873 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:37.968118 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:37.968554 32696 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
W20240501 14:02:37.980520 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 79653.9us
W20240501 14:02:37.980922 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb04ec77da194832aa63d69aba307de3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:37.985867 28744 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:37.986145 28598 catalog_manager.cc:5484] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:37.991804 31061 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:37.993638 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:38.010408 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:37.981917 30928 raft_consensus.cc:1203] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:38.045044 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:38.046718 32667 tablet_copy_client.cc:783] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:38.047034 32667 tablet_copy_client.cc:647] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:38.049108 32667 tablet_copy_client.cc:523] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:38.082433 32667 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:38.178882 32725 ts_tablet_manager.cc:902] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:38.255789 32725 tablet_copy_client.cc:308] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:38.269805 28873 consensus_queue.cc:237] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:38.281127 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 12510f71b6a742ab8cc359243de8119b from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:52377)
I20240501 14:02:38.281363 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 12510f71b6a742ab8cc359243de8119b from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:52377: session id = 26acba62745c48e3ba8180748d55b211-12510f71b6a742ab8cc359243de8119b
I20240501 14:02:38.287099 29027 tablet_copy_source_session.cc:205] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:38.290974 32725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12510f71b6a742ab8cc359243de8119b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:38.301237 29007 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:38.302143 32384 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:38.311908 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:38.320868 32725 tablet_copy_client.cc:783] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:38.324849 32725 tablet_copy_client.cc:647] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:38.329010 32725 tablet_copy_client.cc:523] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:38.350150 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:38.355657 28744 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:38.359428 32384 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:38.382097 28811 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2998b60071cd408aa1cd06fb57d76a8f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:38.384155 32687 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:38.389284 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:38.391718 28743 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:38.396842 32725 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:38.402925 28603 catalog_manager.cc:5484] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:38.408313 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:38.422446 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:38.436018 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:38.448212 28744 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:38.457139 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:38.457327 312 ts_tablet_manager.cc:902] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:38.473548 29137 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:38.474244 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 135da52054584a27959efdd9126cfcef. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:38.474911 32565 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:38.475301 28873 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:38.477128 32565 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:38.478130 29007 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:38.565785 28876 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:38.567198 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:38.563998 32565 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:38.573766 312 tablet_copy_client.cc:308] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:38.575810 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet fb04ec77da194832aa63d69aba307de3 from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:52377)
I20240501 14:02:38.576013 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet fb04ec77da194832aa63d69aba307de3 from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:52377: session id = 26acba62745c48e3ba8180748d55b211-fb04ec77da194832aa63d69aba307de3
I20240501 14:02:38.577795 29137 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:38.583799 29027 tablet_copy_source_session.cc:205] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:38.586537 312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb04ec77da194832aa63d69aba307de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:38.588624 28595 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:38.602027 32629 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:38.615772 32629 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:38.616173 32629 ts_tablet_manager.cc:1367] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.667s user 0.062s sys 0.001s
I20240501 14:02:38.617115 32629 raft_consensus.cc:348] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:38.620048 32629 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:38.621133 32629 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:38.631893 312 tablet_copy_client.cc:783] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:38.632476 312 tablet_copy_client.cc:647] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:38.637775 312 tablet_copy_client.cc:523] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:38.641991 312 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:38.648125 32667 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:38.648581 32667 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:38.648897 32667 ts_tablet_manager.cc:1367] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.567s user 0.054s sys 0.008s
I20240501 14:02:38.658895 32667 raft_consensus.cc:348] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:38.659220 32667 raft_consensus.cc:729] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:38.660351 32667 consensus_queue.cc:260] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:38.670078 32667 ts_tablet_manager.cc:1398] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240501 14:02:38.672426 32629 ts_tablet_manager.cc:1398] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.056s user 0.006s sys 0.005s
I20240501 14:02:38.672422 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-39f1ab3a94db41b08ac90bba52418a93 received from {username='slave'} at 127.16.36.134:55989
I20240501 14:02:38.672636 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-39f1ab3a94db41b08ac90bba52418a93 on tablet 39f1ab3a94db41b08ac90bba52418a93 with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:38.674271 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-e0bf08cbb6d347d783b1fa7762c802df received from {username='slave'} at 127.16.36.135:34125
I20240501 14:02:38.674523 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session a0e901103604414c829b40edf9f24c28-e0bf08cbb6d347d783b1fa7762c802df on tablet e0bf08cbb6d347d783b1fa7762c802df with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:38.677947 32629 ts_tablet_manager.cc:902] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:38.717358 32725 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:38.721701 32629 tablet_copy_client.cc:308] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:38.723381 32725 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:38.723893 32725 ts_tablet_manager.cc:1367] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.327s user 0.061s sys 0.000s
I20240501 14:02:38.729566 32725 raft_consensus.cc:348] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:38.730965 32725 raft_consensus.cc:729] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:38.732048 32725 consensus_queue.cc:260] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:38.739851 32725 ts_tablet_manager.cc:1398] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:02:38.741727 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-12510f71b6a742ab8cc359243de8119b received from {username='slave'} at 127.16.36.133:52377
I20240501 14:02:38.741902 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 26acba62745c48e3ba8180748d55b211-12510f71b6a742ab8cc359243de8119b on tablet 12510f71b6a742ab8cc359243de8119b with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:38.760771 312 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:38.761281 312 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:38.761626 312 ts_tablet_manager.cc:1367] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.120s user 0.051s sys 0.003s
I20240501 14:02:38.762518 312 raft_consensus.cc:348] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:38.762856 312 raft_consensus.cc:729] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:38.763127 312 consensus_queue.cc:260] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:38.763746 312 ts_tablet_manager.cc:1398] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:38.764492 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-fb04ec77da194832aa63d69aba307de3 received from {username='slave'} at 127.16.36.133:52377
I20240501 14:02:38.764672 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 26acba62745c48e3ba8180748d55b211-fb04ec77da194832aa63d69aba307de3 on tablet fb04ec77da194832aa63d69aba307de3 with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:38.813226 30794 raft_consensus.cc:1203] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:38.836591 30662 raft_consensus.cc:1203] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:38.852640 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 2998b60071cd408aa1cd06fb57d76a8f from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34181)
I20240501 14:02:38.852838 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 2998b60071cd408aa1cd06fb57d76a8f from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34181: session id = a0e901103604414c829b40edf9f24c28-2998b60071cd408aa1cd06fb57d76a8f
I20240501 14:02:38.857370 28896 tablet_copy_source_session.cc:205] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:38.859797 32629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998b60071cd408aa1cd06fb57d76a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:38.868563 32629 tablet_copy_client.cc:783] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:38.868880 32629 tablet_copy_client.cc:647] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:38.872793 32629 tablet_copy_client.cc:523] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:38.890466 32629 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:38.898895 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:38.924383 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:38.931720 377 ts_tablet_manager.cc:902] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:38.968006 29007 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:38.977190 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:38.985996 32629 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:38.987037 32629 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:38.987373 32629 ts_tablet_manager.cc:1367] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.107s user 0.057s sys 0.008s
I20240501 14:02:38.991050 32629 raft_consensus.cc:348] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:38.991418 32629 raft_consensus.cc:729] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:38.991775 32629 consensus_queue.cc:260] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:38.996098 32629 ts_tablet_manager.cc:1398] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:02:38.997100 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-2998b60071cd408aa1cd06fb57d76a8f received from {username='slave'} at 127.16.36.135:34181
I20240501 14:02:38.997248 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session a0e901103604414c829b40edf9f24c28-2998b60071cd408aa1cd06fb57d76a8f on tablet 2998b60071cd408aa1cd06fb57d76a8f with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:39.007925 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:39.033098 377 tablet_copy_client.cc:308] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:39.034322 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 135da52054584a27959efdd9126cfcef from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43925)
I20240501 14:02:39.034538 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 135da52054584a27959efdd9126cfcef from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43925: session id = 826ccf8358b6453d88bcb132290b82a5-135da52054584a27959efdd9126cfcef
I20240501 14:02:39.037786 28765 tablet_copy_source_session.cc:205] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:39.047474 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:39.047812 377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135da52054584a27959efdd9126cfcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:39.050419 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:02:39.057155 28941 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec1b3fec785f43c4a3bacedecd8472dd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:39.057165 28744 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.058015 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:39.057912 32255 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.066959 29137 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.067842 32255 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.069304 377 tablet_copy_client.cc:783] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:39.069628 377 tablet_copy_client.cc:647] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:39.081925 377 tablet_copy_client.cc:523] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:39.085703 32696 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.089772 377 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:39.094146 29137 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.118036 30662 raft_consensus.cc:1203] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:39.135813 28744 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.140707 28603 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.169773 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:39.183584 30928 raft_consensus.cc:1203] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:39.198208 30928 raft_consensus.cc:1203] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:39.222731 377 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:39.223454 377 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:39.223963 377 ts_tablet_manager.cc:1367] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.134s user 0.062s sys 0.006s
I20240501 14:02:39.225346 377 raft_consensus.cc:348] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:39.225759 377 raft_consensus.cc:729] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:39.226222 377 consensus_queue.cc:260] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:39.231294 377 ts_tablet_manager.cc:1398] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 14:02:39.232316 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-135da52054584a27959efdd9126cfcef received from {username='slave'} at 127.16.36.136:43925
I20240501 14:02:39.232544 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-135da52054584a27959efdd9126cfcef on tablet 135da52054584a27959efdd9126cfcef with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:39.280665 29137 consensus_queue.cc:237] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:39.296635 28873 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:39.297703 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4402de0b48041c383568feb9365147e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:39.297919 29007 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.298324 438 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.299048 32558 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.323920 31061 raft_consensus.cc:1203] T 135da52054584a27959efdd9126cfcef P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:39.332698 438 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.336479 28873 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.347112 28596 catalog_manager.cc:5484] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.358018 29006 consensus_queue.cc:237] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:39.355960 29007 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.389389 29137 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.389696 28873 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:39.390473 28941 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fdedd39e12a49659d989f37b3287d87. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:39.390726 32696 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.391072 32255 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.438903 28875 consensus_queue.cc:237] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
W20240501 14:02:39.453117 28811 consensus_peers.cc:485] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c0bc741ed064dd7ad8ec70219496c6f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:39.454919 29006 raft_consensus.cc:1261] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.455605 29138 raft_consensus.cc:1261] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.456274 32687 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.456701 311 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.468220 32255 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.493798 29137 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.498327 28596 catalog_manager.cc:5484] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.506382 28873 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:39.540112 32687 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.552906 29006 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.562835 29138 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.562902 28596 catalog_manager.cc:5484] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.611449 29137 consensus_queue.cc:237] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:39.615238 28873 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:02:39.616241 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4402de0b48041c383568feb9365147e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:39.617182 29006 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:02:39.618177 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4402de0b48041c383568feb9365147e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:39.619029 411 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:39.619500 411 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:39.623287 32558 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:39.628252 29006 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:39.639592 28873 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:39.647305 28596 catalog_manager.cc:5484] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:39.699201 494 ts_tablet_manager.cc:902] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:39.762521 494 tablet_copy_client.cc:308] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:39.763468 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet ec1b3fec785f43c4a3bacedecd8472dd from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52677)
I20240501 14:02:39.763670 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet ec1b3fec785f43c4a3bacedecd8472dd from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52677: session id = 8fc5d8222c164a679e7a99966da1eaaf-ec1b3fec785f43c4a3bacedecd8472dd
I20240501 14:02:39.771859 29027 tablet_copy_source_session.cc:205] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:39.777017 494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1b3fec785f43c4a3bacedecd8472dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:39.803205 29006 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
W20240501 14:02:39.807978 28941 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:39.808292 29137 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:39.809674 32255 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:39.821763 28744 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:39.823454 32255 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:39.824330 494 tablet_copy_client.cc:783] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:39.824702 494 tablet_copy_client.cc:647] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:39.835984 494 tablet_copy_client.cc:523] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:39.841960 494 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
W20240501 14:02:39.851439 28942 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec1b3fec785f43c4a3bacedecd8472dd. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:02:39.856272 28678 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:37325, user_credentials={real_user=slave}} blocked reactor thread for 78107.1us
I20240501 14:02:39.877267 540 ts_tablet_manager.cc:902] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:39.885587 32255 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.895166 29137 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.900414 28744 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.902411 28596 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:39.935851 28744 consensus_queue.cc:237] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:39.951081 29137 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.959437 527 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:39.966084 28873 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:39.967106 535 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:40.005645 535 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.007529 540 tablet_copy_client.cc:308] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:40.008561 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 1fdedd39e12a49659d989f37b3287d87 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52677)
I20240501 14:02:40.008757 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 1fdedd39e12a49659d989f37b3287d87 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52677: session id = 8fc5d8222c164a679e7a99966da1eaaf-1fdedd39e12a49659d989f37b3287d87
I20240501 14:02:40.012025 29137 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.013034 29027 tablet_copy_source_session.cc:205] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:40.019541 28596 catalog_manager.cc:5484] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.055121 562 ts_tablet_manager.cc:902] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:40.057703 540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdedd39e12a49659d989f37b3287d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:40.051395 28873 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.116394 494 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:40.116957 494 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:40.117297 494 ts_tablet_manager.cc:1367] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.275s user 0.060s sys 0.000s
I20240501 14:02:40.129487 562 tablet_copy_client.cc:308] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:40.130784 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 7c0bc741ed064dd7ad8ec70219496c6f from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34181)
I20240501 14:02:40.131053 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 7c0bc741ed064dd7ad8ec70219496c6f from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34181: session id = a0e901103604414c829b40edf9f24c28-7c0bc741ed064dd7ad8ec70219496c6f
I20240501 14:02:40.135561 28896 tablet_copy_source_session.cc:205] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:40.142976 606 ts_tablet_manager.cc:902] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:40.145159 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0bc741ed064dd7ad8ec70219496c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:40.145680 494 raft_consensus.cc:348] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:40.148949 494 raft_consensus.cc:729] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:40.150192 494 consensus_queue.cc:260] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:02:40.159787 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 014635e01a22420b8bc21e879449749b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.150900 592 ts_tablet_manager.cc:902] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:40.204797 540 tablet_copy_client.cc:783] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:40.205164 540 tablet_copy_client.cc:647] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:40.206610 592 tablet_copy_client.cc:308] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:40.207720 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet f4402de0b48041c383568feb9365147e from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:55989)
I20240501 14:02:40.207918 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet f4402de0b48041c383568feb9365147e from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:55989: session id = 8fc5d8222c164a679e7a99966da1eaaf-f4402de0b48041c383568feb9365147e
I20240501 14:02:40.212224 29158 tablet_copy_source_session.cc:205] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:40.211825 540 tablet_copy_client.cc:523] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:40.214004 592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4402de0b48041c383568feb9365147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:40.223536 494 ts_tablet_manager.cc:1398] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.106s user 0.003s sys 0.002s
I20240501 14:02:40.224661 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-ec1b3fec785f43c4a3bacedecd8472dd received from {username='slave'} at 127.16.36.134:52677
I20240501 14:02:40.224869 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-ec1b3fec785f43c4a3bacedecd8472dd on tablet ec1b3fec785f43c4a3bacedecd8472dd with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:40.232192 28873 consensus_queue.cc:237] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:40.235224 592 tablet_copy_client.cc:783] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:40.235543 592 tablet_copy_client.cc:647] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:40.237414 592 tablet_copy_client.cc:523] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:40.240521 562 tablet_copy_client.cc:783] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:40.240948 562 tablet_copy_client.cc:647] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:40.241297 592 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:40.272943 562 tablet_copy_client.cc:523] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:40.278685 540 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:40.285066 29006 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:40.286679 628 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:40.291603 562 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
W20240501 14:02:40.294981 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 49513.4us
W20240501 14:02:40.297952 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb712628c7764a138f5425d14b724d9e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.298408 606 tablet_copy_client.cc:308] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:40.299353 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet f4402de0b48041c383568feb9365147e from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:48133)
I20240501 14:02:40.299552 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet f4402de0b48041c383568feb9365147e from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:48133: session id = 26acba62745c48e3ba8180748d55b211-f4402de0b48041c383568feb9365147e
I20240501 14:02:40.303248 29158 tablet_copy_source_session.cc:205] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:40.303640 28744 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:40.304415 32687 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:40.305433 606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4402de0b48041c383568feb9365147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:40.319010 628 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.322163 28744 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.329761 28596 catalog_manager.cc:5484] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.345845 640 ts_tablet_manager.cc:902] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:40.349819 29006 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:40.352077 30794 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:40.394037 606 tablet_copy_client.cc:783] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:40.394487 606 tablet_copy_client.cc:647] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:40.396770 606 tablet_copy_client.cc:523] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:40.411528 592 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:40.414868 592 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:40.415277 592 ts_tablet_manager.cc:1367] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.174s user 0.053s sys 0.014s
I20240501 14:02:40.420305 592 raft_consensus.cc:348] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:40.420704 592 raft_consensus.cc:729] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:40.421030 592 consensus_queue.cc:260] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:40.422340 592 ts_tablet_manager.cc:1398] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:02:40.423300 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-f4402de0b48041c383568feb9365147e received from {username='slave'} at 127.16.36.134:55989
I20240501 14:02:40.423472 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-f4402de0b48041c383568feb9365147e on tablet f4402de0b48041c383568feb9365147e with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:40.442289 606 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:40.471385 540 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:40.471952 540 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:40.472261 540 ts_tablet_manager.cc:1367] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.194s user 0.050s sys 0.012s
I20240501 14:02:40.473141 540 raft_consensus.cc:348] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:40.473403 540 raft_consensus.cc:729] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:40.473662 540 consensus_queue.cc:260] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:40.474408 540 ts_tablet_manager.cc:1398] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:40.477514 562 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:40.478099 562 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:40.478411 562 ts_tablet_manager.cc:1367] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.187s user 0.050s sys 0.017s
I20240501 14:02:40.479529 562 raft_consensus.cc:348] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:40.479918 562 raft_consensus.cc:729] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:40.480235 562 consensus_queue.cc:260] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:40.482754 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-1fdedd39e12a49659d989f37b3287d87 received from {username='slave'} at 127.16.36.134:52677
I20240501 14:02:40.482926 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-1fdedd39e12a49659d989f37b3287d87 on tablet 1fdedd39e12a49659d989f37b3287d87 with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:40.504156 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:40.515451 562 ts_tablet_manager.cc:1398] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.037s user 0.011s sys 0.007s
I20240501 14:02:40.518759 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-7c0bc741ed064dd7ad8ec70219496c6f received from {username='slave'} at 127.16.36.135:34181
I20240501 14:02:40.518927 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session a0e901103604414c829b40edf9f24c28-7c0bc741ed064dd7ad8ec70219496c6f on tablet 7c0bc741ed064dd7ad8ec70219496c6f with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:40.523085 640 tablet_copy_client.cc:308] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:40.555253 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:40.559854 28743 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:40.559842 28744 consensus_queue.cc:237] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:40.566076 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:40.607196 691 ts_tablet_manager.cc:902] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
W20240501 14:02:40.611485 28680 consensus_peers.cc:485] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aaf52ee71b44511b7b8909711ab0a42. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.612874 30928 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:02:40.613046 28679 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0bf08cbb6d347d783b1fa7762c802df. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.613694 535 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:40.614925 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:40.628307 28873 raft_consensus.cc:1261] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:40.628836 29006 raft_consensus.cc:1261] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:40.629173 535 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:40.629914 535 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:40.647465 30794 raft_consensus.cc:1203] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:40.657380 29007 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:40.658100 527 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:40.665838 535 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:40.692943 711 ts_tablet_manager.cc:902] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:40.709772 30794 raft_consensus.cc:1203] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:40.721789 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet ec1b3fec785f43c4a3bacedecd8472dd from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34245)
I20240501 14:02:40.722046 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet ec1b3fec785f43c4a3bacedecd8472dd from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34245: session id = a0e901103604414c829b40edf9f24c28-ec1b3fec785f43c4a3bacedecd8472dd
I20240501 14:02:40.725286 29027 tablet_copy_source_session.cc:205] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:40.733286 640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1b3fec785f43c4a3bacedecd8472dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:40.738596 711 tablet_copy_client.cc:308] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:40.739641 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet bb712628c7764a138f5425d14b724d9e from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:49903)
I20240501 14:02:40.739846 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet bb712628c7764a138f5425d14b724d9e from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:49903: session id = 26acba62745c48e3ba8180748d55b211-bb712628c7764a138f5425d14b724d9e
I20240501 14:02:40.743392 28896 tablet_copy_source_session.cc:205] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:40.745034 711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb712628c7764a138f5425d14b724d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:40.764806 30928 raft_consensus.cc:1203] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:40.765200 711 tablet_copy_client.cc:783] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:40.765579 711 tablet_copy_client.cc:647] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:40.768010 711 tablet_copy_client.cc:523] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:40.769251 29137 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:40.770442 723 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:40.774057 711 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:40.785072 30928 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:40.785900 28596 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:40.829468 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:40.831354 606 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:40.831952 606 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:40.832342 606 ts_tablet_manager.cc:1367] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.390s user 0.101s sys 0.063s
I20240501 14:02:40.833524 606 raft_consensus.cc:348] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:40.834064 606 raft_consensus.cc:729] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:40.834432 606 consensus_queue.cc:260] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:40.836033 691 tablet_copy_client.cc:308] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:40.838203 606 ts_tablet_manager.cc:1398] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:02:40.840440 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-f4402de0b48041c383568feb9365147e received from {username='slave'} at 127.16.36.133:48133
I20240501 14:02:40.840600 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 26acba62745c48e3ba8180748d55b211-f4402de0b48041c383568feb9365147e on tablet f4402de0b48041c383568feb9365147e with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:40.846777 29137 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:40.852692 29006 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:40.853432 707 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:40.855791 640 tablet_copy_client.cc:783] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:40.856102 640 tablet_copy_client.cc:647] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:40.856086 29005 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:40.865676 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:40.873400 640 tablet_copy_client.cc:523] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:40.877652 28596 catalog_manager.cc:5484] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:40.906224 28873 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:40.943614 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 014635e01a22420b8bc21e879449749b from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:32915)
I20240501 14:02:40.943840 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 014635e01a22420b8bc21e879449749b from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:32915: session id = 26acba62745c48e3ba8180748d55b211-014635e01a22420b8bc21e879449749b
I20240501 14:02:40.946849 28765 tablet_copy_source_session.cc:205] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:40.948887 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014635e01a22420b8bc21e879449749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:40.956586 691 tablet_copy_client.cc:783] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:40.957074 691 tablet_copy_client.cc:647] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:40.889139 640 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:40.966056 29006 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
W20240501 14:02:40.973860 28941 consensus_peers.cc:485] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8b9db597e654ed48b679abf58b8a2c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:40.975354 28743 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:40.976092 32255 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:40.978502 28873 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:40.979099 32255 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:40.990692 691 tablet_copy_client.cc:523] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:40.993022 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:41.006490 691 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:41.029224 32255 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:41.035806 28743 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:41.036535 711 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:41.037499 711 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:41.046360 28595 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:41.037791 711 ts_tablet_manager.cc:1367] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.266s user 0.063s sys 0.014s
I20240501 14:02:41.051149 711 raft_consensus.cc:348] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:41.051498 711 raft_consensus.cc:729] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:41.051790 711 consensus_queue.cc:260] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:41.052402 711 ts_tablet_manager.cc:1398] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:02:41.053359 28873 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:41.076182 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-bb712628c7764a138f5425d14b724d9e received from {username='slave'} at 127.16.36.133:49903
I20240501 14:02:41.076365 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 26acba62745c48e3ba8180748d55b211-bb712628c7764a138f5425d14b724d9e on tablet bb712628c7764a138f5425d14b724d9e with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:41.130007 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:41.133965 775 ts_tablet_manager.cc:902] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:41.177098 775 tablet_copy_client.cc:308] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:41.177949 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 6aaf52ee71b44511b7b8909711ab0a42 from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34125)
I20240501 14:02:41.178133 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 6aaf52ee71b44511b7b8909711ab0a42 from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34125: session id = a0e901103604414c829b40edf9f24c28-6aaf52ee71b44511b7b8909711ab0a42
I20240501 14:02:41.181440 28765 tablet_copy_source_session.cc:205] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:41.183930 775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aaf52ee71b44511b7b8909711ab0a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:41.199878 29137 consensus_queue.cc:237] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:41.213747 800 ts_tablet_manager.cc:902] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:41.244976 28873 consensus_queue.cc:237] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:41.256238 775 tablet_copy_client.cc:783] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:41.260941 775 tablet_copy_client.cc:647] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:41.262291 28743 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:41.263011 486 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:41.264042 775 tablet_copy_client.cc:523] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:41.281874 775 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:41.253055 30661 raft_consensus.cc:1203] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:41.228148 691 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:41.287998 30927 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.288079 691 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:41.291224 800 tablet_copy_client.cc:308] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:41.292224 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet e0bf08cbb6d347d783b1fa7762c802df from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43925)
I20240501 14:02:41.292456 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet e0bf08cbb6d347d783b1fa7762c802df from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43925: session id = 826ccf8358b6453d88bcb132290b82a5-e0bf08cbb6d347d783b1fa7762c802df
I20240501 14:02:41.296492 28765 tablet_copy_source_session.cc:205] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:41.298501 800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bf08cbb6d347d783b1fa7762c802df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:41.298882 640 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:41.298923 29006 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:41.299412 640 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:41.299786 640 ts_tablet_manager.cc:1367] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.411s user 0.057s sys 0.000s
I20240501 14:02:41.300632 640 raft_consensus.cc:348] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:41.300912 640 raft_consensus.cc:729] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:41.301098 486 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:41.302336 691 ts_tablet_manager.cc:1367] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.308s user 0.064s sys 0.011s
I20240501 14:02:41.303385 633 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20240501 14:02:41.309971 691 raft_consensus.cc:348] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
W20240501 14:02:41.314061 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b6a54ac32694640bffdf1c7f3b035cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:41.315958 691 raft_consensus.cc:729] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:41.316972 640 consensus_queue.cc:260] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:41.318080 691 consensus_queue.cc:260] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:41.319211 640 ts_tablet_manager.cc:1398] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240501 14:02:41.324819 691 ts_tablet_manager.cc:1398] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:02:41.326038 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-014635e01a22420b8bc21e879449749b received from {username='slave'} at 127.16.36.133:32915
I20240501 14:02:41.326180 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 26acba62745c48e3ba8180748d55b211-014635e01a22420b8bc21e879449749b on tablet 014635e01a22420b8bc21e879449749b with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:41.328457 29005 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.329209 633 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:41.322263 30928 raft_consensus.cc:1203] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:41.337934 30661 raft_consensus.cc:1203] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:41.341790 486 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:41.348021 29007 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:41.351621 29005 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.359577 633 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.361562 28596 catalog_manager.cc:5484] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:41.369485 28596 catalog_manager.cc:5484] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.382421 800 tablet_copy_client.cc:783] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:41.384656 800 tablet_copy_client.cc:647] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:41.388216 800 tablet_copy_client.cc:523] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:41.401340 800 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:41.419975 28743 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.424791 830 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:41.426836 28745 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
W20240501 14:02:41.435446 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 67650us
I20240501 14:02:41.438323 30928 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.446076 775 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:41.449383 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-ec1b3fec785f43c4a3bacedecd8472dd received from {username='slave'} at 127.16.36.135:34245
I20240501 14:02:41.449546 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session a0e901103604414c829b40edf9f24c28-ec1b3fec785f43c4a3bacedecd8472dd on tablet ec1b3fec785f43c4a3bacedecd8472dd with peer a0e901103604414c829b40edf9f24c28
W20240501 14:02:41.456998 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2998b60071cd408aa1cd06fb57d76a8f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:41.466996 775 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:41.467442 775 ts_tablet_manager.cc:1367] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.198s user 0.055s sys 0.003s
I20240501 14:02:41.468370 775 raft_consensus.cc:348] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:41.468639 775 raft_consensus.cc:729] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:41.468884 775 consensus_queue.cc:260] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:41.469583 775 ts_tablet_manager.cc:1398] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:02:41.471264 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-6aaf52ee71b44511b7b8909711ab0a42 received from {username='slave'} at 127.16.36.135:34125
I20240501 14:02:41.471413 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session a0e901103604414c829b40edf9f24c28-6aaf52ee71b44511b7b8909711ab0a42 on tablet 6aaf52ee71b44511b7b8909711ab0a42 with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:41.481915 28743 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.535095 30928 raft_consensus.cc:1203] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:41.581996 858 ts_tablet_manager.cc:902] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:41.589715 30662 raft_consensus.cc:1203] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:41.599299 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:41.632664 858 tablet_copy_client.cc:308] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:41.636979 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet e8b9db597e654ed48b679abf58b8a2c8 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43429)
I20240501 14:02:41.637195 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet e8b9db597e654ed48b679abf58b8a2c8 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43429: session id = 826ccf8358b6453d88bcb132290b82a5-e8b9db597e654ed48b679abf58b8a2c8
I20240501 14:02:41.641155 29027 tablet_copy_source_session.cc:205] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:41.655453 800 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:41.656051 800 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:41.656409 800 ts_tablet_manager.cc:1367] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.255s user 0.057s sys 0.013s
I20240501 14:02:41.657506 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:41.657451 800 raft_consensus.cc:348] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:41.661013 800 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:41.661618 800 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:41.668538 800 ts_tablet_manager.cc:1398] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240501 14:02:41.671927 858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b9db597e654ed48b679abf58b8a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:41.673691 29005 consensus_queue.cc:237] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:41.676436 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-e0bf08cbb6d347d783b1fa7762c802df received from {username='slave'} at 127.16.36.136:43925
I20240501 14:02:41.676580 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-e0bf08cbb6d347d783b1fa7762c802df on tablet e0bf08cbb6d347d783b1fa7762c802df with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:41.678272 30662 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:02:41.679157 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12510f71b6a742ab8cc359243de8119b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:41.679394 32255 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:41.681844 28743 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.682648 709 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:41.687912 709 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:41.693614 28873 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.695425 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:41.695318 32255 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:41.697860 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:41.699774 28743 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:41.719612 30662 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:41.727070 28595 catalog_manager.cc:5484] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 reported cstate change: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:41.734213 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:02:41.742913 28940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 47571.1us
I20240501 14:02:41.756268 28873 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:41.775784 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:41.797691 28873 consensus_queue.cc:237] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:41.803006 30662 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.803782 830 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:41.805317 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb712628c7764a138f5425d14b724d9e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:41.806191 28743 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.806872 831 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:41.810662 29005 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:41.812356 831 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:41.823514 858 tablet_copy_client.cc:783] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:41.823900 858 tablet_copy_client.cc:647] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:41.829414 858 tablet_copy_client.cc:523] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:41.832531 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:41.846102 831 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.854285 28745 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.854709 30662 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.855633 28596 catalog_manager.cc:5484] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.859846 29005 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:41.874922 800 ts_tablet_manager.cc:902] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:41.878062 858 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:41.886219 928 ts_tablet_manager.cc:902] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:41.908937 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:41.950824 800 tablet_copy_client.cc:308] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:41.975483 928 tablet_copy_client.cc:308] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:41.996905 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 2998b60071cd408aa1cd06fb57d76a8f from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52893)
I20240501 14:02:41.997133 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 2998b60071cd408aa1cd06fb57d76a8f from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52893: session id = 8fc5d8222c164a679e7a99966da1eaaf-2998b60071cd408aa1cd06fb57d76a8f
I20240501 14:02:42.000049 28896 tablet_copy_source_session.cc:205] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:42.001171 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 6b6a54ac32694640bffdf1c7f3b035cc from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:52585)
I20240501 14:02:42.001349 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 6b6a54ac32694640bffdf1c7f3b035cc from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:52585: session id = 826ccf8358b6453d88bcb132290b82a5-6b6a54ac32694640bffdf1c7f3b035cc
I20240501 14:02:42.004321 29158 tablet_copy_source_session.cc:205] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:42.006353 800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6a54ac32694640bffdf1c7f3b035cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:42.006599 928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2998b60071cd408aa1cd06fb57d76a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:42.013746 800 tablet_copy_client.cc:783] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:42.014003 800 tablet_copy_client.cc:647] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:42.017315 928 tablet_copy_client.cc:783] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:42.017714 928 tablet_copy_client.cc:647] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:42.017709 800 tablet_copy_client.cc:523] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:42.019935 928 tablet_copy_client.cc:523] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:42.023913 928 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:42.033913 800 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:42.046530 28745 consensus_queue.cc:237] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:42.070704 858 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:42.072155 858 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:42.072492 858 ts_tablet_manager.cc:1367] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.195s user 0.071s sys 0.010s
I20240501 14:02:42.075183 858 raft_consensus.cc:348] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:42.075471 858 raft_consensus.cc:729] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:42.075788 858 consensus_queue.cc:260] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:42.079346 858 ts_tablet_manager.cc:1398] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240501 14:02:42.080479 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-e8b9db597e654ed48b679abf58b8a2c8 received from {username='slave'} at 127.16.36.136:43429
I20240501 14:02:42.080679 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-e8b9db597e654ed48b679abf58b8a2c8 on tablet e8b9db597e654ed48b679abf58b8a2c8 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:42.099422 29137 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:42.098568 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 763b2e84afa64924ad5eeeeb3265a1d8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:42.101457 925 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:42.109347 28873 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:42.110738 800 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:42.111240 800 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:42.111241 29005 consensus_queue.cc:237] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:42.111598 800 ts_tablet_manager.cc:1367] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.078s user 0.059s sys 0.002s
I20240501 14:02:42.112602 800 raft_consensus.cc:348] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:42.113003 800 raft_consensus.cc:729] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:42.113135 925 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:42.113319 800 consensus_queue.cc:260] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:42.114102 800 ts_tablet_manager.cc:1398] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:02:42.114909 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-6b6a54ac32694640bffdf1c7f3b035cc received from {username='slave'} at 127.16.36.136:52585
I20240501 14:02:42.115048 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-6b6a54ac32694640bffdf1c7f3b035cc on tablet 6b6a54ac32694640bffdf1c7f3b035cc with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:42.121001 928 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:42.121464 928 tablet_bootstrap.cc:492] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:42.121812 928 ts_tablet_manager.cc:1367] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.098s user 0.049s sys 0.010s
I20240501 14:02:42.127836 28745 raft_consensus.cc:1261] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:02:42.128036 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9bb3eaf3f0e40f8902b9426a845d9b2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:42.129118 907 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:42.129245 29138 raft_consensus.cc:1261] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:42.132467 31060 raft_consensus.cc:1203] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:42.134146 928 raft_consensus.cc:348] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:42.134538 928 raft_consensus.cc:729] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:42.134928 928 consensus_queue.cc:260] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:42.135514 709 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:42.144053 928 ts_tablet_manager.cc:1398] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
I20240501 14:02:42.145074 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-2998b60071cd408aa1cd06fb57d76a8f received from {username='slave'} at 127.16.36.134:52893
I20240501 14:02:42.145247 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-2998b60071cd408aa1cd06fb57d76a8f on tablet 2998b60071cd408aa1cd06fb57d76a8f with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:42.174345 907 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.178508 29136 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.183517 28595 catalog_manager.cc:5484] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.185395 925 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.193114 29138 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.214380 31060 raft_consensus.cc:1203] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:42.221220 29005 consensus_queue.cc:237] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:42.237018 29137 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.237679 28873 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.238791 30795 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.240018 909 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.240586 909 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.244915 909 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:42.245536 28942 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fdedd39e12a49659d989f37b3287d87. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:42.249249 28876 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.255872 28595 catalog_manager.cc:5484] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.263900 28744 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.279951 993 ts_tablet_manager.cc:902] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:42.287773 907 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.295069 28876 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.295301 28595 catalog_manager.cc:5484] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.295178 29136 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.312333 30795 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:42.370863 30795 raft_consensus.cc:1203] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:42.392218 928 ts_tablet_manager.cc:902] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:42.433295 928 tablet_copy_client.cc:308] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:42.434412 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet bb712628c7764a138f5425d14b724d9e from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52893)
I20240501 14:02:42.434640 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet bb712628c7764a138f5425d14b724d9e from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52893: session id = 8fc5d8222c164a679e7a99966da1eaaf-bb712628c7764a138f5425d14b724d9e
I20240501 14:02:42.439877 28896 tablet_copy_source_session.cc:205] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:42.442009 928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb712628c7764a138f5425d14b724d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:42.444837 993 tablet_copy_client.cc:308] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:42.445706 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 12510f71b6a742ab8cc359243de8119b from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34245)
I20240501 14:02:42.445891 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 12510f71b6a742ab8cc359243de8119b from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34245: session id = a0e901103604414c829b40edf9f24c28-12510f71b6a742ab8cc359243de8119b
I20240501 14:02:42.448791 29027 tablet_copy_source_session.cc:205] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:42.451258 993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12510f71b6a742ab8cc359243de8119b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:42.472055 31060 raft_consensus.cc:1203] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:42.488126 928 tablet_copy_client.cc:783] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:42.488526 928 tablet_copy_client.cc:647] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:42.491096 928 tablet_copy_client.cc:523] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:42.491873 993 tablet_copy_client.cc:783] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:42.492157 993 tablet_copy_client.cc:647] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:42.495445 993 tablet_copy_client.cc:523] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:42.500916 928 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:42.519809 993 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:42.527623 28873 consensus_queue.cc:237] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:42.541339 30662 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.542657 29005 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.542587 830 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.542831 28744 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.543745 830 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:42.545348 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e821b93aa9b84ff3a968860a5b33b918. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:42.546455 830 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.596458 1070 ts_tablet_manager.cc:902] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:42.600260 831 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:42.603199 30661 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:42.611647 28596 catalog_manager.cc:5484] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:42.613359 28745 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:42.632288 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:42.645370 29007 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:42.698290 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:42.732230 29005 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:42.742820 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:42.744061 1095 ts_tablet_manager.cc:902] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:42.748332 1070 tablet_copy_client.cc:308] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:42.749749 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 763b2e84afa64924ad5eeeeb3265a1d8 from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:32915)
I20240501 14:02:42.749940 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 763b2e84afa64924ad5eeeeb3265a1d8 from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:32915: session id = 26acba62745c48e3ba8180748d55b211-763b2e84afa64924ad5eeeeb3265a1d8
I20240501 14:02:42.750075 1094 ts_tablet_manager.cc:902] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:42.752920 28765 tablet_copy_source_session.cc:205] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:42.759197 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:02:42.759505 28942 consensus_peers.cc:485] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8b9db597e654ed48b679abf58b8a2c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:42.760746 31060 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.761631 709 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.763339 28873 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.767326 709 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.772156 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:42.784238 1070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b2e84afa64924ad5eeeeb3265a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:42.789541 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:42.819613 928 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:42.820170 928 tablet_bootstrap.cc:492] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:42.820492 928 ts_tablet_manager.cc:1367] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.320s user 0.058s sys 0.005s
I20240501 14:02:42.821477 928 raft_consensus.cc:348] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:42.829097 1095 tablet_copy_client.cc:308] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:42.828964 907 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:42.821743 928 raft_consensus.cc:729] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:42.833276 928 consensus_queue.cc:260] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:42.836421 928 ts_tablet_manager.cc:1398] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.016s user 0.003s sys 0.001s
I20240501 14:02:42.836021 31061 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:42.838825 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-bb712628c7764a138f5425d14b724d9e received from {username='slave'} at 127.16.36.134:52893
I20240501 14:02:42.838986 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-bb712628c7764a138f5425d14b724d9e on tablet bb712628c7764a138f5425d14b724d9e with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:42.843057 1094 tablet_copy_client.cc:308] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:42.843360 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet e9bb3eaf3f0e40f8902b9426a845d9b2 from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:52377)
I20240501 14:02:42.843538 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet e9bb3eaf3f0e40f8902b9426a845d9b2 from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:52377: session id = 26acba62745c48e3ba8180748d55b211-e9bb3eaf3f0e40f8902b9426a845d9b2
I20240501 14:02:42.845530 29026 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 1fdedd39e12a49659d989f37b3287d87 from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:52377)
I20240501 14:02:42.845757 29026 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 1fdedd39e12a49659d989f37b3287d87 from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:52377: session id = 26acba62745c48e3ba8180748d55b211-1fdedd39e12a49659d989f37b3287d87
I20240501 14:02:42.848974 29026 tablet_copy_source_session.cc:205] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:42.851748 28596 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 reported cstate change: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:42.858502 28876 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:42.863607 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:42.868603 29136 consensus_queue.cc:237] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:42.874361 29027 tablet_copy_source_session.cc:205] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:42.874912 1094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdedd39e12a49659d989f37b3287d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:42.881130 1095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bb3eaf3f0e40f8902b9426a845d9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:42.887208 29005 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:02:42.887686 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b6a54ac32694640bffdf1c7f3b035cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:42.888218 486 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.890599 28745 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.892246 907 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.892536 1094 tablet_copy_client.cc:783] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:42.893110 1094 tablet_copy_client.cc:647] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:42.894328 1095 tablet_copy_client.cc:783] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:42.894577 1095 tablet_copy_client.cc:647] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:42.895418 31061 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.896107 1094 tablet_copy_client.cc:523] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:42.896092 486 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.896216 28745 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:42.897293 1095 tablet_copy_client.cc:523] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:42.897248 486 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:42.901638 1095 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:42.910578 993 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:42.911127 993 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:42.911439 993 ts_tablet_manager.cc:1367] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.392s user 0.046s sys 0.019s
I20240501 14:02:42.912274 993 raft_consensus.cc:348] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:42.912559 993 raft_consensus.cc:729] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:42.912804 993 consensus_queue.cc:260] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:42.916076 1094 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
W20240501 14:02:42.921885 28941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 75865us
I20240501 14:02:42.932555 993 ts_tablet_manager.cc:1398] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240501 14:02:42.943672 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-12510f71b6a742ab8cc359243de8119b received from {username='slave'} at 127.16.36.135:34245
I20240501 14:02:42.943892 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session a0e901103604414c829b40edf9f24c28-12510f71b6a742ab8cc359243de8119b on tablet 12510f71b6a742ab8cc359243de8119b with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:42.970929 28744 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:42.991250 486 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.000553 29005 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.002276 31061 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.011096 1070 tablet_copy_client.cc:783] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:43.010876 28598 catalog_manager.cc:5484] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.011381 1070 tablet_copy_client.cc:647] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:43.019834 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:43.027981 1070 tablet_copy_client.cc:523] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:43.033919 28745 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.051823 1070 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:43.101935 1162 ts_tablet_manager.cc:902] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:43.116802 1094 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:43.119045 1094 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:43.122920 1094 ts_tablet_manager.cc:1367] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.207s user 0.055s sys 0.002s
I20240501 14:02:43.123947 1094 raft_consensus.cc:348] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.127916 1094 raft_consensus.cc:729] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:43.129621 1095 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:43.128338 1094 consensus_queue.cc:260] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.130162 1095 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:43.133682 1094 ts_tablet_manager.cc:1398] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:02:43.135329 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-1fdedd39e12a49659d989f37b3287d87 received from {username='slave'} at 127.16.36.133:52377
I20240501 14:02:43.135485 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 26acba62745c48e3ba8180748d55b211-1fdedd39e12a49659d989f37b3287d87 on tablet 1fdedd39e12a49659d989f37b3287d87 with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:43.143117 1095 ts_tablet_manager.cc:1367] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.242s user 0.051s sys 0.014s
I20240501 14:02:43.145272 1162 tablet_copy_client.cc:308] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:43.146116 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet e821b93aa9b84ff3a968860a5b33b918 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:60541)
I20240501 14:02:43.146312 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet e821b93aa9b84ff3a968860a5b33b918 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:60541: session id = 826ccf8358b6453d88bcb132290b82a5-e821b93aa9b84ff3a968860a5b33b918
I20240501 14:02:43.149564 28896 tablet_copy_source_session.cc:205] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:43.152451 1162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e821b93aa9b84ff3a968860a5b33b918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:43.156589 1095 raft_consensus.cc:348] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.156905 1095 raft_consensus.cc:729] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:43.161636 1095 consensus_queue.cc:260] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.162379 1095 ts_tablet_manager.cc:1398] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240501 14:02:43.163096 1162 tablet_copy_client.cc:783] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:43.163472 1162 tablet_copy_client.cc:647] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:43.165444 1162 tablet_copy_client.cc:523] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:43.171873 1162 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:43.186512 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-e9bb3eaf3f0e40f8902b9426a845d9b2 received from {username='slave'} at 127.16.36.133:52377
I20240501 14:02:43.186728 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 26acba62745c48e3ba8180748d55b211-e9bb3eaf3f0e40f8902b9426a845d9b2 on tablet e9bb3eaf3f0e40f8902b9426a845d9b2 with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:43.189097 28745 consensus_queue.cc:237] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:43.192564 28873 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:02:43.194262 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:43.194465 925 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:43.211876 29136 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:43.212725 925 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:43.221258 30795 raft_consensus.cc:1203] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:43.257390 1070 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:43.257958 1070 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:43.258303 1070 ts_tablet_manager.cc:1367] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.207s user 0.047s sys 0.015s
I20240501 14:02:43.258811 30662 raft_consensus.cc:1203] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:43.259215 1070 raft_consensus.cc:348] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.259478 1070 raft_consensus.cc:729] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:43.259723 1070 consensus_queue.cc:260] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.260316 1070 ts_tablet_manager.cc:1398] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:02:43.269208 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-763b2e84afa64924ad5eeeeb3265a1d8 received from {username='slave'} at 127.16.36.133:32915
I20240501 14:02:43.269460 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 26acba62745c48e3ba8180748d55b211-763b2e84afa64924ad5eeeeb3265a1d8 on tablet 763b2e84afa64924ad5eeeeb3265a1d8 with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:43.271222 925 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.281163 29136 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.291527 28596 catalog_manager.cc:5484] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.298959 28873 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
W20240501 14:02:43.313681 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 763b2e84afa64924ad5eeeeb3265a1d8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:43.317238 28745 consensus_queue.cc:237] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:43.325042 30928 raft_consensus.cc:1203] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:43.334200 30662 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:43.335597 1202 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:43.336416 29136 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:43.337779 28873 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:43.337548 1204 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:02:43.338860 925 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:02:43.344163 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 014635e01a22420b8bc21e879449749b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:43.374526 1202 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:43.377853 1162 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:43.378433 1162 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:43.378813 1162 ts_tablet_manager.cc:1367] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.208s user 0.059s sys 0.013s
I20240501 14:02:43.379890 1162 raft_consensus.cc:348] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:43.379793 28876 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:43.380254 1162 raft_consensus.cc:729] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:43.380630 1162 consensus_queue.cc:260] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:43.387372 993 ts_tablet_manager.cc:902] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:43.389691 1162 ts_tablet_manager.cc:1398] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20240501 14:02:43.393854 28595 catalog_manager.cc:5484] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:43.399201 30662 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:43.402588 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-e821b93aa9b84ff3a968860a5b33b918 received from {username='slave'} at 127.16.36.136:60541
I20240501 14:02:43.402782 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-e821b93aa9b84ff3a968860a5b33b918 on tablet e821b93aa9b84ff3a968860a5b33b918 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:43.423853 29137 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:43.453747 31061 raft_consensus.cc:1203] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:43.463761 993 tablet_copy_client.cc:308] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:43.464625 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet e8b9db597e654ed48b679abf58b8a2c8 from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34245)
I20240501 14:02:43.464805 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet e8b9db597e654ed48b679abf58b8a2c8 from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34245: session id = a0e901103604414c829b40edf9f24c28-e8b9db597e654ed48b679abf58b8a2c8
I20240501 14:02:43.467801 29027 tablet_copy_source_session.cc:205] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:43.471731 993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b9db597e654ed48b679abf58b8a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:43.476249 29137 consensus_queue.cc:237] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.483305 993 tablet_copy_client.cc:783] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:43.484747 993 tablet_copy_client.cc:647] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:43.486759 993 tablet_copy_client.cc:523] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:43.491829 993 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
W20240501 14:02:43.498149 29073 consensus_peers.cc:485] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b7c7b963588436bb778e45da469c7d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:43.498262 28873 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:43.499109 827 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:43.506537 28745 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:43.507354 827 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:43.508281 28876 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:02:43.526961 28810 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63b7f2a8d8a84faf86c4de16b3cc6663. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:43.527606 29007 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:43.528259 830 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:43.529162 29137 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:43.530906 830 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:43.535038 827 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.545290 28745 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.546733 28595 catalog_manager.cc:5484] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.542197 28876 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.560640 1248 ts_tablet_manager.cc:902] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:43.560587 817 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.561429 29007 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.566942 29137 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.572806 28598 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.617022 29137 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:43.619450 993 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:43.619918 993 tablet_bootstrap.cc:492] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:43.620169 993 ts_tablet_manager.cc:1367] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.128s user 0.049s sys 0.014s
I20240501 14:02:43.626245 993 raft_consensus.cc:348] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:43.626555 993 raft_consensus.cc:729] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:43.628294 993 consensus_queue.cc:260] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:43.630256 993 ts_tablet_manager.cc:1398] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:02:43.631163 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-e8b9db597e654ed48b679abf58b8a2c8 received from {username='slave'} at 127.16.36.135:34245
I20240501 14:02:43.631292 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session a0e901103604414c829b40edf9f24c28-e8b9db597e654ed48b679abf58b8a2c8 on tablet e8b9db597e654ed48b679abf58b8a2c8 with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:43.638641 30928 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:02:43.638978 29073 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de161b10c84f4bcc85f8cf7b199f99db. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:43.640020 486 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:43.640260 28745 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:43.640931 486 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:43.642788 28876 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:43.643563 486 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:43.643801 30662 raft_consensus.cc:1203] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:43.680306 486 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.683547 28744 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
W20240501 14:02:43.687393 1273 log.cc:924] Time spent T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240501 14:02:43.688333 30928 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.692685 1270 ts_tablet_manager.cc:902] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:43.694205 28595 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.697919 28876 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:43.726473 1248 tablet_copy_client.cc:308] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:43.731818 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 6b6a54ac32694640bffdf1c7f3b035cc from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:55989)
I20240501 14:02:43.732004 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 6b6a54ac32694640bffdf1c7f3b035cc from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:55989: session id = 8fc5d8222c164a679e7a99966da1eaaf-6b6a54ac32694640bffdf1c7f3b035cc
I20240501 14:02:43.735414 29158 tablet_copy_source_session.cc:205] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:43.737967 30662 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:43.739382 1248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6a54ac32694640bffdf1c7f3b035cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:43.755199 1270 tablet_copy_client.cc:308] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:43.793279 1248 tablet_copy_client.cc:783] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:43.793622 1248 tablet_copy_client.cc:647] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:43.797385 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 763b2e84afa64924ad5eeeeb3265a1d8 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:60347)
I20240501 14:02:43.797681 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 763b2e84afa64924ad5eeeeb3265a1d8 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:60347: session id = 8fc5d8222c164a679e7a99966da1eaaf-763b2e84afa64924ad5eeeeb3265a1d8
I20240501 14:02:43.803972 28765 tablet_copy_source_session.cc:205] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:43.804792 1248 tablet_copy_client.cc:523] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:43.805761 1270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b2e84afa64924ad5eeeeb3265a1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:43.832091 1270 tablet_copy_client.cc:783] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:43.832861 1270 tablet_copy_client.cc:647] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:43.835063 1270 tablet_copy_client.cc:523] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:43.845455 1248 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:43.848630 1270 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:43.865674 1162 ts_tablet_manager.cc:902] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:43.914937 30928 raft_consensus.cc:1203] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:43.955636 1248 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:43.958370 1248 tablet_bootstrap.cc:492] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:43.959163 1248 ts_tablet_manager.cc:1367] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.113s user 0.062s sys 0.004s
I20240501 14:02:43.960125 1248 raft_consensus.cc:348] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:43.960412 1248 raft_consensus.cc:729] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:43.961254 1248 consensus_queue.cc:260] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:43.963336 29137 consensus_queue.cc:237] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:43.968606 1248 ts_tablet_manager.cc:1398] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240501 14:02:43.969509 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-6b6a54ac32694640bffdf1c7f3b035cc received from {username='slave'} at 127.16.36.134:55989
I20240501 14:02:43.969663 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-6b6a54ac32694640bffdf1c7f3b035cc on tablet 6b6a54ac32694640bffdf1c7f3b035cc with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:43.975883 30662 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:43.976377 30795 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:43.977048 29007 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:43.977725 486 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:43.978457 486 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:43.980814 28876 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:43.983597 827 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:43.984338 827 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:43.989389 1270 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:43.989915 1270 tablet_bootstrap.cc:492] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:43.990227 1270 ts_tablet_manager.cc:1367] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.142s user 0.055s sys 0.006s
W20240501 14:02:43.990417 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4402de0b48041c383568feb9365147e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:43.991127 1270 raft_consensus.cc:348] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:43.991398 1270 raft_consensus.cc:729] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:43.991696 1270 consensus_queue.cc:260] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:43.992420 1270 ts_tablet_manager.cc:1398] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:02:43.993264 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-763b2e84afa64924ad5eeeeb3265a1d8 received from {username='slave'} at 127.16.36.134:60347
I20240501 14:02:43.993423 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-763b2e84afa64924ad5eeeeb3265a1d8 on tablet 763b2e84afa64924ad5eeeeb3265a1d8 with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:44.003952 486 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:44.012887 30662 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:44.027655 29007 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:44.029796 30795 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:44.030138 28596 catalog_manager.cc:5484] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 reported cstate change: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:44.060825 1367 ts_tablet_manager.cc:902] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:44.077615 1162 tablet_copy_client.cc:308] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:44.078616 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 014635e01a22420b8bc21e879449749b from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43925)
I20240501 14:02:44.078868 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 014635e01a22420b8bc21e879449749b from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43925: session id = 826ccf8358b6453d88bcb132290b82a5-014635e01a22420b8bc21e879449749b
I20240501 14:02:44.082031 28765 tablet_copy_source_session.cc:205] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:44.084890 28876 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:44.086902 1162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014635e01a22420b8bc21e879449749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:44.112797 1367 tablet_copy_client.cc:308] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:44.116053 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet de161b10c84f4bcc85f8cf7b199f99db from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:48133)
I20240501 14:02:44.116325 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet de161b10c84f4bcc85f8cf7b199f99db from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:48133: session id = 26acba62745c48e3ba8180748d55b211-de161b10c84f4bcc85f8cf7b199f99db
I20240501 14:02:44.118018 1162 tablet_copy_client.cc:783] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:44.118389 1162 tablet_copy_client.cc:647] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:44.120062 29158 tablet_copy_source_session.cc:205] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:44.121811 1367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de161b10c84f4bcc85f8cf7b199f99db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:44.122167 1162 tablet_copy_client.cc:523] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:44.126308 1372 ts_tablet_manager.cc:902] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:44.126825 1162 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:44.134992 1367 tablet_copy_client.cc:783] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:44.135337 1367 tablet_copy_client.cc:647] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:44.144517 1270 ts_tablet_manager.cc:902] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:44.161758 1367 tablet_copy_client.cc:523] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:44.162720 30795 raft_consensus.cc:1203] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:44.180009 1367 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:44.199200 1372 tablet_copy_client.cc:308] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:44.200068 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 9b7c7b963588436bb778e45da469c7d5 from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:48133)
I20240501 14:02:44.200302 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 9b7c7b963588436bb778e45da469c7d5 from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:48133: session id = 26acba62745c48e3ba8180748d55b211-9b7c7b963588436bb778e45da469c7d5
I20240501 14:02:44.204762 29158 tablet_copy_source_session.cc:205] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:44.206379 1372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7c7b963588436bb778e45da469c7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:44.214457 1372 tablet_copy_client.cc:783] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:44.214776 1372 tablet_copy_client.cc:647] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:44.217181 1372 tablet_copy_client.cc:523] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:44.222295 1372 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:02:44.236883 29007 consensus_queue.cc:237] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:02:44.275969 28941 consensus_peers.cc:485] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d8c435795a04b1b90b6e2ff914676c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:44.277436 29137 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:44.279692 28876 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:02:44.280050 1336 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:02:44.297334 1336 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:02:44.300410 28679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 75733.2us
I20240501 14:02:44.311676 1270 tablet_copy_client.cc:308] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:44.312824 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52893)
I20240501 14:02:44.313048 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 63b7f2a8d8a84faf86c4de16b3cc6663 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52893: session id = 8fc5d8222c164a679e7a99966da1eaaf-63b7f2a8d8a84faf86c4de16b3cc6663
I20240501 14:02:44.316113 28896 tablet_copy_source_session.cc:205] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:44.318037 1270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7f2a8d8a84faf86c4de16b3cc6663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:44.329676 1270 tablet_copy_client.cc:783] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:44.330258 1270 tablet_copy_client.cc:647] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:44.332473 1336 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:44.335430 29136 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:44.347760 28596 catalog_manager.cc:5484] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:44.351593 28876 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:44.354478 1270 tablet_copy_client.cc:523] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:44.359820 1270 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:44.471175 30795 raft_consensus.cc:1203] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:44.490414 1372 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:44.492790 1162 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:44.493288 1162 tablet_bootstrap.cc:492] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:44.493575 1162 ts_tablet_manager.cc:1367] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.368s user 0.114s sys 0.064s
I20240501 14:02:44.494479 1162 raft_consensus.cc:348] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:44.494808 1162 raft_consensus.cc:729] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:44.494891 1367 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:44.495105 1162 consensus_queue.cc:260] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:44.495362 1367 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:44.495725 1367 ts_tablet_manager.cc:1367] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.316s user 0.064s sys 0.011s
I20240501 14:02:44.497684 1372 tablet_bootstrap.cc:492] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:02:44.498050 1372 ts_tablet_manager.cc:1367] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.276s user 0.059s sys 0.006s
I20240501 14:02:44.499162 1372 raft_consensus.cc:348] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:44.499513 1372 raft_consensus.cc:729] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:44.506856 1372 consensus_queue.cc:260] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:44.507807 1372 ts_tablet_manager.cc:1398] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:02:44.508649 1367 raft_consensus.cc:348] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:44.508989 1367 raft_consensus.cc:729] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:02:44.509292 1367 consensus_queue.cc:260] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:44.509857 1367 ts_tablet_manager.cc:1398] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:02:44.510115 1162 ts_tablet_manager.cc:1398] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.016s user 0.008s sys 0.004s
I20240501 14:02:44.514758 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-014635e01a22420b8bc21e879449749b received from {username='slave'} at 127.16.36.136:43925
I20240501 14:02:44.514958 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-014635e01a22420b8bc21e879449749b on tablet 014635e01a22420b8bc21e879449749b with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:44.543150 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-de161b10c84f4bcc85f8cf7b199f99db received from {username='slave'} at 127.16.36.133:48133
I20240501 14:02:44.543341 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 26acba62745c48e3ba8180748d55b211-de161b10c84f4bcc85f8cf7b199f99db on tablet de161b10c84f4bcc85f8cf7b199f99db with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:44.544517 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-9b7c7b963588436bb778e45da469c7d5 received from {username='slave'} at 127.16.36.133:48133
I20240501 14:02:44.544652 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 26acba62745c48e3ba8180748d55b211-9b7c7b963588436bb778e45da469c7d5 on tablet 9b7c7b963588436bb778e45da469c7d5 with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:02:44.555521 1270 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:44.556085 1270 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:44.556455 1270 ts_tablet_manager.cc:1367] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.197s user 0.061s sys 0.005s
I20240501 14:02:44.557785 1270 raft_consensus.cc:348] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:44.558140 1270 raft_consensus.cc:729] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:44.558466 1270 consensus_queue.cc:260] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:44.567421 1270 ts_tablet_manager.cc:1398] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240501 14:02:44.568843 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-63b7f2a8d8a84faf86c4de16b3cc6663 received from {username='slave'} at 127.16.36.134:52893
I20240501 14:02:44.569046 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-63b7f2a8d8a84faf86c4de16b3cc6663 on tablet 63b7f2a8d8a84faf86c4de16b3cc6663 with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:44.571626 1162 ts_tablet_manager.cc:902] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:44.587790 29007 consensus_queue.cc:237] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:44.598487 30795 raft_consensus.cc:1203] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:44.620558 30662 raft_consensus.cc:1203] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:44.634613 30662 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:02:44.635526 28942 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fdedd39e12a49659d989f37b3287d87. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:44.635812 1336 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:44.645761 29137 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:44.645761 30795 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:44.646481 1132 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:44.647023 30660 raft_consensus.cc:1203] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:44.647197 1132 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:44.657866 28876 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:44.658672 1336 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:44.664911 1336 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:44.669559 28876 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:44.673061 30660 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:44.679426 1162 tablet_copy_client.cc:308] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:02:44.678495 28595 catalog_manager.cc:5484] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 6, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:44.680238 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet f4402de0b48041c383568feb9365147e from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:52585)
I20240501 14:02:44.680447 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet f4402de0b48041c383568feb9365147e from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:52585: session id = 826ccf8358b6453d88bcb132290b82a5-f4402de0b48041c383568feb9365147e
I20240501 14:02:44.686223 29158 tablet_copy_source_session.cc:205] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:44.689802 1162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4402de0b48041c383568feb9365147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:44.696663 1270 ts_tablet_manager.cc:902] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:44.708024 1162 tablet_copy_client.cc:783] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:44.708408 1162 tablet_copy_client.cc:647] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:44.710510 1162 tablet_copy_client.cc:523] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:44.718531 1162 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:02:44.730000 30795 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:44.735646 29137 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:44.751847 1270 tablet_copy_client.cc:308] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:44.752753 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 4d8c435795a04b1b90b6e2ff914676c8 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52677)
I20240501 14:02:44.752972 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 4d8c435795a04b1b90b6e2ff914676c8 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52677: session id = 8fc5d8222c164a679e7a99966da1eaaf-4d8c435795a04b1b90b6e2ff914676c8
I20240501 14:02:44.757179 29027 tablet_copy_source_session.cc:205] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:44.759440 1270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8c435795a04b1b90b6e2ff914676c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:44.802995 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:44.805131 1270 tablet_copy_client.cc:783] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:44.805799 1270 tablet_copy_client.cc:647] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:44.813010 1270 tablet_copy_client.cc:523] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:44.828303 1270 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
I20240501 14:02:44.843344 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:44.868732 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:44.880169 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:44.888303 31060 raft_consensus.cc:1203] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:02:44.901986 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:44.934249 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:44.935401 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:44.992348 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:45.023125 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:45.050179 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.052592 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.074276 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.091331 1270 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:45.168249 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.112298 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.169677 1270 tablet_bootstrap.cc:492] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:02:45.176290 1270 ts_tablet_manager.cc:1367] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.348s user 0.059s sys 0.071s
I20240501 14:02:45.171695 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.198187 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:45.206022 1548 ts_tablet_manager.cc:902] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:02:45.206312 1270 raft_consensus.cc:348] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:45.236915 1270 raft_consensus.cc:729] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:02:45.237409 1270 consensus_queue.cc:260] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:45.238494 1270 ts_tablet_manager.cc:1398] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.043s user 0.001s sys 0.002s
I20240501 14:02:45.239862 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-4d8c435795a04b1b90b6e2ff914676c8 received from {username='slave'} at 127.16.36.134:52677
I20240501 14:02:45.240036 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-4d8c435795a04b1b90b6e2ff914676c8 on tablet 4d8c435795a04b1b90b6e2ff914676c8 with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:45.274757 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.277442 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.279510 1548 tablet_copy_client.cc:308] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:02:45.281301 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 1fdedd39e12a49659d989f37b3287d87 from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34245)
I20240501 14:02:45.281499 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 1fdedd39e12a49659d989f37b3287d87 from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34245: session id = a0e901103604414c829b40edf9f24c28-1fdedd39e12a49659d989f37b3287d87
I20240501 14:02:45.284556 29027 tablet_copy_source_session.cc:205] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:45.287982 1548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdedd39e12a49659d989f37b3287d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:45.303514 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:45.324024 1548 tablet_copy_client.cc:783] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:45.324950 1548 tablet_copy_client.cc:647] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:45.343681 1548 tablet_copy_client.cc:523] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:45.354660 1548 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:45.396560 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:45.400282 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.400527 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:45.429358 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:45.487432 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.515077 1162 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:45.518693 1162 tablet_bootstrap.cc:492] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:02:45.519085 1162 ts_tablet_manager.cc:1367] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.801s user 0.052s sys 0.017s
I20240501 14:02:45.525677 1162 raft_consensus.cc:348] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:45.526051 1162 raft_consensus.cc:729] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:02:45.532074 1162 consensus_queue.cc:260] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:45.542546 1548 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:45.543587 1548 tablet_bootstrap.cc:492] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:45.544713 1548 ts_tablet_manager.cc:1367] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.189s user 0.044s sys 0.012s
I20240501 14:02:45.545735 1548 raft_consensus.cc:348] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:45.550936 1162 ts_tablet_manager.cc:1398] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.032s user 0.002s sys 0.001s
I20240501 14:02:45.556710 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-f4402de0b48041c383568feb9365147e received from {username='slave'} at 127.16.36.136:52585
I20240501 14:02:45.556869 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-f4402de0b48041c383568feb9365147e on tablet f4402de0b48041c383568feb9365147e with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:45.589047 1548 raft_consensus.cc:729] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:45.589493 1548 consensus_queue.cc:260] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:45.607057 30795 raft_consensus.cc:1203] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:02:45.621415 31060 raft_consensus.cc:1203] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:02:45.660869 1548 ts_tablet_manager.cc:1398] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.116s user 0.011s sys 0.011s
I20240501 14:02:45.662107 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-1fdedd39e12a49659d989f37b3287d87 received from {username='slave'} at 127.16.36.135:34245
I20240501 14:02:45.662236 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session a0e901103604414c829b40edf9f24c28-1fdedd39e12a49659d989f37b3287d87 on tablet 1fdedd39e12a49659d989f37b3287d87 with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:45.718513 30928 raft_consensus.cc:1203] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:02:45.897379 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:45.954195 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:46.018193 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:46.047030 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:46.047362 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.112874 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.137686 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:46.142285 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.413668 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.436897 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:46.477401 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.535670 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:02:46.585481 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 78389.4us
I20240501 14:02:46.589120 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:46.610842 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:46.682153 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.731909 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:46.932912 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:46.944262 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.969484 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:46.972601 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:47.025362 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.025830 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.033121 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:47.056694 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:47.121917 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:47.123193 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.140043 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:47.142931 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:47.184461 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.205197 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.236019 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.329644 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:47.813648 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.835460 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:47.864195 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:47.905829 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:47.919598 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.930124 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.966853 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.980075 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:47.988420 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:48.038717 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:48.098130 29138 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:48.115238 29006 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:48.142889 30927 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:48.143572 1817 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:48.143817 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:48.148541 31060 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:48.148541 28873 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:48.149549 1817 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:48.173260 1817 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:48.174674 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:48.187582 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:48.201256 30662 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:48.201696 30927 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:48.202114 1778 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:48.204334 1778 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:48.209304 28876 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:48.211352 1778 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:48.219173 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:48.231613 1336 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.235961 28876 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.238233 30928 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.246206 28603 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.260423 31061 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.266175 1366 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:48.275818 30927 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:48.276230 30662 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:48.285199 28875 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:48.286852 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:48.310487 28596 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:48.327512 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b9db597e654ed48b679abf58b8a2c8 with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:48.327796 28723 tablet_service.cc:1514] Processing DeleteTablet for tablet de161b10c84f4bcc85f8cf7b199f99db with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:48.368486 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:48.390653 1951 tablet_replica.cc:330] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:48.391309 1951 raft_consensus.cc:2229] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:48.391726 1951 raft_consensus.cc:2258] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:48.393607 1951 ts_tablet_manager.cc:1875] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:48.487679 1951 ts_tablet_manager.cc:1888] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:48.490466 1951 log.cc:1196] T e8b9db597e654ed48b679abf58b8a2c8 P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/e8b9db597e654ed48b679abf58b8a2c8
I20240501 14:02:48.491808 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet e8b9db597e654ed48b679abf58b8a2c8 (table rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]) successfully deleted
I20240501 14:02:48.493986 1951 tablet_replica.cc:330] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:48.494294 1951 raft_consensus.cc:2229] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:48.494868 1951 raft_consensus.cc:2258] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:48.499913 1951 ts_tablet_manager.cc:1875] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:48.578913 1951 ts_tablet_manager.cc:1888] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:48.579968 1951 log.cc:1196] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/de161b10c84f4bcc85f8cf7b199f99db
I20240501 14:02:48.600092 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:48.604887 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet de161b10c84f4bcc85f8cf7b199f99db (table rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]) successfully deleted
I20240501 14:02:48.628022 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:48.652678 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:48.660283 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:48.735474 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:48.798326 28875 consensus_queue.cc:237] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:48.811712 30927 raft_consensus.cc:1261] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:48.812502 29138 raft_consensus.cc:1261] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:48.812644 1812 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:48.813225 1638 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:48.820271 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:48.836236 1638 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.840533 30928 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.847544 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:48.854197 28595 catalog_manager.cc:5484] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.888335 29136 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:48.931507 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:49.040192 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0bc741ed064dd7ad8ec70219496c6f with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59004
I20240501 14:02:49.058769 2064 tablet_replica.cc:330] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:02:49.067332 2064 raft_consensus.cc:2229] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:49.068708 2064 raft_consensus.cc:2258] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:49.070484 2064 ts_tablet_manager.cc:1875] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:49.117034 2064 ts_tablet_manager.cc:1888] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:49.132114 2064 log.cc:1196] T 7c0bc741ed064dd7ad8ec70219496c6f P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/7c0bc741ed064dd7ad8ec70219496c6f
I20240501 14:02:49.133996 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 7c0bc741ed064dd7ad8ec70219496c6f (table rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]) successfully deleted
I20240501 14:02:49.159672 28875 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:49.163755 29136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93"
dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51534
I20240501 14:02:49.164112 29136 raft_consensus.cc:595] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Received request to transfer leadership
I20240501 14:02:49.197695 30795 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:49.198457 1638 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:49.206687 29136 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:49.207912 1638 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:49.208662 1638 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:49.209379 30795 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:49.218246 28596 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:49.225344 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59004
I20240501 14:02:49.225849 2064 tablet_replica.cc:330] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:02:49.226085 2064 raft_consensus.cc:2229] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:49.226459 2064 raft_consensus.cc:2258] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:49.227330 2064 ts_tablet_manager.cc:1875] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:49.240190 2064 ts_tablet_manager.cc:1888] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:49.241212 2064 log.cc:1196] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/63b7f2a8d8a84faf86c4de16b3cc6663
I20240501 14:02:49.245375 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 63b7f2a8d8a84faf86c4de16b3cc6663 (table rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]) successfully deleted
I20240501 14:02:49.249805 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:49.319955 29117 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:49.321084 29136 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:49.329519 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:49.365690 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:49.385331 1778 raft_consensus.cc:979] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: : Instructing follower 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:02:49.385529 1778 raft_consensus.cc:1067] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Signalling peer 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:02:49.386973 30794 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93"
dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf"
from {username='slave'} at 127.16.36.132:59495
I20240501 14:02:49.387313 30794 raft_consensus.cc:482] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:02:49.387526 30794 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:49.391883 30794 raft_consensus.cc:504] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:49.396800 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:02:49.397102 28745 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:49.401762 28745 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:02:49.404906 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:02:49.405157 29007 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:49.409770 29007 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:02:49.410133 29006 raft_consensus.cc:1226] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Rejecting Update request from peer 3a3972b170c54344a4ba61f176f837ba for earlier term 1. Current term is 2. Ops: []
I20240501 14:02:49.410881 1778 consensus_queue.cc:1046] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:49.411955 1366 raft_consensus.cc:3032] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:02:49.412109 1366 raft_consensus.cc:729] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:02:49.412469 1366 consensus_queue.cc:260] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:49.412909 1366 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:02:49.432305 30729 leader_election.cc:302] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8, 8fc5d8222c164a679e7a99966da1eaaf; no voters:
I20240501 14:02:49.432902 2096 raft_consensus.cc:2781] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:02:49.434531 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:02:49.439606 29136 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:02:49.441011 30775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:49.441035 30794 leader_election.cc:288] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:02:49.444864 2096 raft_consensus.cc:686] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Becoming Leader. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Running, Role: LEADER
I20240501 14:02:49.445557 2096 consensus_queue.cc:237] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:49.460954 28596 catalog_manager.cc:5484] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: term changed from 1 to 2, leader changed from 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) to 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134). New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:49.476807 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:49.526947 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:49.529676 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:49.531700 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:49.548415 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:49.557302 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:49.606364 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:49.634752 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:49.640699 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:49.643675 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:49.669016 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:49.889322 28745 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:49.890393 2096 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:49.939388 29007 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:49.940692 2096 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:50.021981 29136 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:02:50.023382 2096 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:50.038336 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:50.063421 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:50.110299 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:50.121331 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:50.138151 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:50.181599 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:02:50.211403 28941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 93735.4us
I20240501 14:02:50.250316 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:02:50.285025 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 99398.1us
I20240501 14:02:50.311465 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:50.327615 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:50.348690 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:50.368151 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:50.425446 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:50.429790 28745 consensus_queue.cc:237] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:50.445654 29007 consensus_queue.cc:237] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:50.456504 30660 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:50.458729 1934 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:50.463120 31061 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:50.463819 1934 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:50.471571 28875 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:50.472383 1935 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:50.493530 1935 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:50.499516 31060 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:50.507881 30660 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:50.515621 28873 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:50.516288 28596 catalog_manager.cc:5484] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:50.541396 30928 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:50.541709 30660 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:50.542748 2221 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:50.543489 2221 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:50.546248 30793 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:50.546942 29136 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:50.547792 1937 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:50.549675 1817 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:50.579984 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:50.595820 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:50.615614 2221 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:50.620362 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:50.629331 30662 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:50.645769 30928 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:50.651966 29117 tablet_service.cc:1514] Processing DeleteTablet for tablet 014635e01a22420b8bc21e879449749b with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50834
I20240501 14:02:50.653694 28598 catalog_manager.cc:5484] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 reported cstate change: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:50.669960 2294 tablet_replica.cc:330] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:02:50.674877 2294 raft_consensus.cc:2229] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:50.677462 2294 raft_consensus.cc:2258] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:50.678962 29138 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:50.684526 30794 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:50.685096 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:50.686342 2294 ts_tablet_manager.cc:1875] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:50.706861 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdedd39e12a49659d989f37b3287d87 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37770
I20240501 14:02:50.737505 2298 tablet_replica.cc:330] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:02:50.738281 2298 raft_consensus.cc:2229] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:50.738395 2294 ts_tablet_manager.cc:1888] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:50.744899 2294 log.cc:1196] T 014635e01a22420b8bc21e879449749b P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/014635e01a22420b8bc21e879449749b
I20240501 14:02:50.747177 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 014635e01a22420b8bc21e879449749b (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
I20240501 14:02:50.762975 2298 raft_consensus.cc:2258] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:50.805496 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:50.816324 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:50.822106 2298 ts_tablet_manager.cc:1875] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:50.832358 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:50.900069 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:50.904393 2298 ts_tablet_manager.cc:1888] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:02:50.904629 2298 log.cc:1196] T 1fdedd39e12a49659d989f37b3287d87 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/1fdedd39e12a49659d989f37b3287d87
I20240501 14:02:50.925285 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 1fdedd39e12a49659d989f37b3287d87 (table rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]) successfully deleted
I20240501 14:02:50.926012 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:51.025368 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:51.052857 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:51.098978 29138 consensus_queue.cc:237] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:51.112475 30660 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:51.113327 2231 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:51.116199 28875 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:51.118115 2231 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:51.139158 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:51.140259 2231 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:51.141638 30662 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:51.143908 28875 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:51.153610 28604 catalog_manager.cc:5484] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:51.158470 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7c7b963588436bb778e45da469c7d5 with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:51.163134 2349 tablet_replica.cc:330] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:51.163491 2349 raft_consensus.cc:2229] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:51.164160 2349 raft_consensus.cc:2258] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:51.165490 2349 ts_tablet_manager.cc:1875] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:51.221156 2349 ts_tablet_manager.cc:1888] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:51.223280 2349 log.cc:1196] T 9b7c7b963588436bb778e45da469c7d5 P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b7c7b963588436bb778e45da469c7d5
I20240501 14:02:51.224527 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet 9b7c7b963588436bb778e45da469c7d5 (table rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]) successfully deleted
I20240501 14:02:51.489363 30793 consensus_queue.cc:237] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:51.503526 28744 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:51.504465 29007 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:02:51.504734 2196 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:51.505580 2196 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:51.563849 2420 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:51.566588 28745 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:51.573133 28595 catalog_manager.cc:5484] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 4 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:51.578011 29117 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f1ab3a94db41b08ac90bba52418a93 with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50834
I20240501 14:02:51.581451 29007 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:51.605708 2425 tablet_replica.cc:330] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:02:51.607514 2425 raft_consensus.cc:2229] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:51.607796 2425 raft_consensus.cc:2258] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:51.609095 2425 ts_tablet_manager.cc:1875] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:51.633308 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:51.657734 2425 ts_tablet_manager.cc:1888] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:02:51.657958 2425 log.cc:1196] T 39f1ab3a94db41b08ac90bba52418a93 P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/39f1ab3a94db41b08ac90bba52418a93
I20240501 14:02:51.676653 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 39f1ab3a94db41b08ac90bba52418a93 (table rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]) successfully deleted
I20240501 14:02:51.680675 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:51.753165 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:51.786036 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:51.805130 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:51.820926 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:51.914793 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:51.928581 28875 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:51.940558 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:51.982453 29136 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:51.986235 1812 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:51.992275 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:52.000118 31060 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:52.002022 1812 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:52.002966 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:52.043601 2306 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:52.048003 31060 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:52.059967 28604 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:52.068931 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet 6feda9b53e484e7ea40a5890b7d51ac4 with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:52.098912 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:52.170444 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:52.197620 2491 tablet_replica.cc:330] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:52.198033 2491 raft_consensus.cc:2229] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 14:02:52.206655 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 212725us
I20240501 14:02:52.223037 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:52.229955 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:52.242038 2491 raft_consensus.cc:2258] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:52.246901 2491 ts_tablet_manager.cc:1875] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:52.269657 29136 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:52.305997 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:52.319522 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:52.344434 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:52.358983 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:52.366200 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:52.392726 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:52.395901 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:52.433786 2491 ts_tablet_manager.cc:1888] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:52.433985 2491 log.cc:1196] T 6feda9b53e484e7ea40a5890b7d51ac4 P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/6feda9b53e484e7ea40a5890b7d51ac4
I20240501 14:02:52.434747 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:52.448370 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:52.452232 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:52.471344 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet 6feda9b53e484e7ea40a5890b7d51ac4 (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
W20240501 14:02:52.839329 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 47240.7us
I20240501 14:02:52.990022 28745 consensus_queue.cc:237] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:52.994493 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:52.996022 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:53.010726 30926 raft_consensus.cc:1261] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:53.010957 28875 raft_consensus.cc:1261] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:53.011706 1935 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:53.012378 1935 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:53.026017 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:53.042661 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:53.101388 2471 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.106065 28873 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.119475 28598 catalog_manager.cc:5484] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.137890 30926 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.157027 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aaf52ee71b44511b7b8909711ab0a42 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59004
W20240501 14:02:53.175424 28678 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:37325, user_credentials={real_user=slave}} blocked reactor thread for 66297.8us
I20240501 14:02:53.212855 2619 tablet_replica.cc:330] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:02:53.213279 2619 raft_consensus.cc:2229] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:53.215487 2619 raft_consensus.cc:2258] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:53.216413 2619 ts_tablet_manager.cc:1875] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:53.237035 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:53.261237 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:53.335278 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:53.337368 28873 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:53.367867 2619 ts_tablet_manager.cc:1888] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:53.374254 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:53.374949 31061 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:53.375926 2610 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:53.389750 2610 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:53.397019 31059 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:53.401217 28598 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:53.406023 29117 tablet_service.cc:1514] Processing DeleteTablet for tablet 6feda9b53e484e7ea40a5890b7d51ac4 with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50834
I20240501 14:02:53.439044 28875 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:53.436081 29007 consensus_queue.cc:237] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
W20240501 14:02:53.450387 28811 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6feda9b53e484e7ea40a5890b7d51ac4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:53.451653 31059 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:53.452426 1812 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:53.456205 2619 log.cc:1196] T 6aaf52ee71b44511b7b8909711ab0a42 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/6aaf52ee71b44511b7b8909711ab0a42
I20240501 14:02:53.459678 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 6aaf52ee71b44511b7b8909711ab0a42 (table rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]) successfully deleted
I20240501 14:02:53.480185 30660 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:53.481372 30926 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:53.481420 2474 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:53.483103 2655 tablet_replica.cc:330] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:02:53.483352 2655 raft_consensus.cc:2229] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:53.483599 2655 raft_consensus.cc:2258] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:53.484100 2549 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:53.484409 2655 ts_tablet_manager.cc:1875] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:53.486261 28875 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:53.486559 1812 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.487047 2474 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:53.491613 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6feda9b53e484e7ea40a5890b7d51ac4 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:02:53.493285 31059 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.502751 28595 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 6 to 7, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.510025 2655 ts_tablet_manager.cc:1888] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:53.510181 2655 log.cc:1196] T 6feda9b53e484e7ea40a5890b7d51ac4 P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/6feda9b53e484e7ea40a5890b7d51ac4
I20240501 14:02:53.513070 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 6feda9b53e484e7ea40a5890b7d51ac4 (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
I20240501 14:02:53.534415 2474 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.540194 28873 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.555557 30926 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.556305 28595 catalog_manager.cc:5484] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.562968 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet 12510f71b6a742ab8cc359243de8119b with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:53.559970 30660 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:53.635772 2672 tablet_replica.cc:330] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:53.636164 2672 raft_consensus.cc:2229] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:53.636538 2672 raft_consensus.cc:2258] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:53.637650 2672 ts_tablet_manager.cc:1875] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:53.731451 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:53.738565 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:53.761970 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:53.766995 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:53.814879 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:53.822050 2672 ts_tablet_manager.cc:1888] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:53.823052 2672 log.cc:1196] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/12510f71b6a742ab8cc359243de8119b
I20240501 14:02:53.845042 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:53.846989 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet 12510f71b6a742ab8cc359243de8119b (table rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]) successfully deleted
I20240501 14:02:53.850946 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:53.857843 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:53.970269 2730 ts_tablet_manager.cc:902] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:02:54.030186 2730 tablet_copy_client.cc:308] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:02:54.031129 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:34181)
I20240501 14:02:54.031327 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:34181: session id = a0e901103604414c829b40edf9f24c28-6feda9b53e484e7ea40a5890b7d51ac4
I20240501 14:02:54.034396 28896 tablet_copy_source_session.cc:205] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:54.041433 2730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6feda9b53e484e7ea40a5890b7d51ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:54.062006 2730 tablet_copy_client.cc:783] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:54.063983 2730 tablet_copy_client.cc:647] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:54.067701 2730 tablet_copy_client.cc:523] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:54.077203 2730 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:02:54.173727 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:54.192790 2730 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:54.193609 2730 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:02:54.194101 2730 ts_tablet_manager.cc:1367] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.119s user 0.080s sys 0.001s
I20240501 14:02:54.195390 2730 raft_consensus.cc:348] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:54.195839 2730 raft_consensus.cc:729] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:02:54.196209 2730 consensus_queue.cc:260] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:54.204684 2730 ts_tablet_manager.cc:1398] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20240501 14:02:54.207382 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-6feda9b53e484e7ea40a5890b7d51ac4 received from {username='slave'} at 127.16.36.135:34181
I20240501 14:02:54.207577 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session a0e901103604414c829b40edf9f24c28-6feda9b53e484e7ea40a5890b7d51ac4 on tablet 6feda9b53e484e7ea40a5890b7d51ac4 with peer a0e901103604414c829b40edf9f24c28
I20240501 14:02:54.332290 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:54.368124 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:54.369941 29007 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:54.387282 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:54.389544 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:54.408190 30926 raft_consensus.cc:1203] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:02:54.420991 30926 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:54.421856 2474 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:54.428969 29136 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:54.429836 2474 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:54.431557 30795 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:54.432291 2474 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:54.451462 2667 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:54.456068 29138 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:54.465267 30795 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:54.464913 28598 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:54.485504 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:54.486900 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:54.501742 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:54.522171 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:54.562988 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:54.581517 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:54.587303 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:54.588840 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:54.610317 30928 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:54.614171 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:54.626319 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:54.651618 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:54.675530 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1b3fec785f43c4a3bacedecd8472dd with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:54.710307 2832 tablet_replica.cc:330] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:54.710963 2832 raft_consensus.cc:2229] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:54.712790 2832 raft_consensus.cc:2258] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:54.714298 2832 ts_tablet_manager.cc:1875] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:54.744807 2832 ts_tablet_manager.cc:1888] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:54.745056 2832 log.cc:1196] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/ec1b3fec785f43c4a3bacedecd8472dd
I20240501 14:02:54.745903 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet ec1b3fec785f43c4a3bacedecd8472dd (table rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]) successfully deleted
I20240501 14:02:54.867988 28744 consensus_queue.cc:237] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:54.889875 30795 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:54.890916 1935 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:54.900185 30660 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:54.901139 1935 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:54.901588 28875 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:54.902392 1935 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:54.930924 1935 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:54.936303 28873 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:54.941515 30662 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:54.953401 28601 catalog_manager.cc:5484] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:54.960119 30795 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:54.964625 29117 tablet_service.cc:1514] Processing DeleteTablet for tablet 763b2e84afa64924ad5eeeeb3265a1d8 with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50834
I20240501 14:02:54.998121 2860 tablet_replica.cc:330] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:02:54.998440 2860 raft_consensus.cc:2229] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:54.998857 2860 raft_consensus.cc:2258] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:55.034097 2860 ts_tablet_manager.cc:1875] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:55.053344 2860 ts_tablet_manager.cc:1888] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:55.053717 2860 log.cc:1196] T 763b2e84afa64924ad5eeeeb3265a1d8 P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/763b2e84afa64924ad5eeeeb3265a1d8
I20240501 14:02:55.058473 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 763b2e84afa64924ad5eeeeb3265a1d8 (table rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]) successfully deleted
I20240501 14:02:55.227520 29136 consensus_queue.cc:237] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:55.246460 30660 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:55.247354 30926 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:55.248445 2826 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:55.249281 29007 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:55.249732 2825 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:55.250496 2825 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:55.285499 2825 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:55.289505 30660 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:55.295338 29007 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:55.300546 28595 catalog_manager.cc:5484] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:55.304304 30926 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:55.312363 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet f516f01e242d4b8789130201f748f3ea with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37770
I20240501 14:02:55.331804 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:55.351167 2928 tablet_replica.cc:330] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:02:55.351833 2928 raft_consensus.cc:2229] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:55.352454 2928 raft_consensus.cc:2258] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:55.353678 2928 ts_tablet_manager.cc:1875] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:55.359545 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:55.364871 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:55.411077 2928 ts_tablet_manager.cc:1888] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:55.413064 2928 log.cc:1196] T f516f01e242d4b8789130201f748f3ea P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/f516f01e242d4b8789130201f748f3ea
I20240501 14:02:55.422020 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:55.422578 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet f516f01e242d4b8789130201f748f3ea (table rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]) successfully deleted
I20240501 14:02:55.447777 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:55.483961 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:55.502183 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:55.512383 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:55.551908 29007 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:55.602671 30795 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:55.603025 28744 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:55.603768 2474 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:55.604491 2474 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:55.642324 2667 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:55.646911 30793 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:55.648145 28744 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:55.655508 28601 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:55.662937 29117 tablet_service.cc:1514] Processing DeleteTablet for tablet efe7fed8e5d44e8a80ffbdc27619a74d with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50834
I20240501 14:02:55.663882 2969 tablet_replica.cc:330] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:02:55.664124 2969 raft_consensus.cc:2229] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:55.664364 2969 raft_consensus.cc:2258] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:55.665202 2969 ts_tablet_manager.cc:1875] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:55.678012 2969 ts_tablet_manager.cc:1888] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:55.678158 2969 log.cc:1196] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/efe7fed8e5d44e8a80ffbdc27619a74d
I20240501 14:02:55.679090 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet efe7fed8e5d44e8a80ffbdc27619a74d (table rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]) successfully deleted
I20240501 14:02:55.684518 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:55.739780 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:55.785080 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:55.787740 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:55.838708 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:55.861053 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:55.915324 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:55.928205 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:56.177798 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:56.216102 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:56.281847 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:56.339880 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:56.382767 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:56.393963 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:56.433959 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:56.451153 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:56.636962 29136 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:56.701339 30926 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:56.701417 28744 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:56.702886 2826 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:56.704128 2826 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:56.738517 2826 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:56.740278 30926 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:56.747874 28604 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:56.780426 28744 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
W20240501 14:02:56.863550 3110 log.cc:924] Time spent T 00000000000000000000000000000000 P 1bdce3de4d694d6c98f909a5b3c450ce: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20240501 14:02:56.883397 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b3db79ff754e8bbe9e5865f7104312 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37770
I20240501 14:02:56.901782 3136 tablet_replica.cc:330] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:02:56.904881 3136 raft_consensus.cc:2229] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:56.906476 3136 raft_consensus.cc:2258] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:56.908717 3136 ts_tablet_manager.cc:1875] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:56.934178 3136 ts_tablet_manager.cc:1888] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:56.937276 3136 log.cc:1196] T 60b3db79ff754e8bbe9e5865f7104312 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/60b3db79ff754e8bbe9e5865f7104312
I20240501 14:02:56.945740 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 60b3db79ff754e8bbe9e5865f7104312 (table rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]) successfully deleted
I20240501 14:02:56.988435 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.025566 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:57.033119 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.046026 28723 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:57.048296 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:57.061997 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:57.088145 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:57.176609 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:57.185770 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.252089 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.307613 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:57.311057 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:57.315297 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:57.327572 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:57.329522 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.331688 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:57.378468 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:57.416222 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.421590 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:57.439321 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:57.455263 28745 consensus_queue.cc:237] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:57.458915 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.460824 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.496158 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:57.506937 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:57.539453 30928 raft_consensus.cc:1261] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:57.541015 3004 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:57.565500 3009 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:57.571827 30928 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:57.574532 28603 catalog_manager.cc:5484] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:57.664773 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aaf52ee71b44511b7b8909711ab0a42 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37770
I20240501 14:02:57.672835 28745 consensus_queue.cc:237] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:57.675781 30928 raft_consensus.cc:1261] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:02:57.676124 28678 consensus_peers.cc:485] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 -> Peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Couldn't send request to peer 3a3972b170c54344a4ba61f176f837ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aaf52ee71b44511b7b8909711ab0a42. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:57.676609 3009 consensus_queue.cc:1035] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:57.679302 3009 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:57.684037 30926 raft_consensus.cc:2932] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:57.685726 3215 tablet_replica.cc:330] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:02:57.686067 3215 raft_consensus.cc:2229] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:57.686331 3215 raft_consensus.cc:2258] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:57.688068 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6aaf52ee71b44511b7b8909711ab0a42 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:02:57.688331 3215 ts_tablet_manager.cc:1875] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:57.692606 28595 catalog_manager.cc:5484] T 6aaf52ee71b44511b7b8909711ab0a42 P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 6 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:02:57.770152 3215 ts_tablet_manager.cc:1888] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:57.779958 3215 log.cc:1196] T 6aaf52ee71b44511b7b8909711ab0a42 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/6aaf52ee71b44511b7b8909711ab0a42
I20240501 14:02:57.783975 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 6aaf52ee71b44511b7b8909711ab0a42 (table rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]) successfully deleted
I20240501 14:02:58.046361 29136 consensus_queue.cc:237] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:58.073498 28873 consensus_queue.cc:237] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:02:58.085809 30793 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:58.086524 3005 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:58.099222 31061 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:58.100011 3005 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:58.119966 29006 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:58.121228 3005 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:58.136950 3092 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:58.139072 30662 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:58.139796 3223 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:58.148128 31059 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:58.149595 28745 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:58.151629 3222 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:58.152549 3222 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:58.169495 31061 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:58.173225 29006 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:58.173879 28595 catalog_manager.cc:5484] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:58.174015 30793 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:02:58.179159 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6a54ac32694640bffdf1c7f3b035cc with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:58.219699 3310 tablet_replica.cc:330] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:58.220002 3310 raft_consensus.cc:2229] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:58.220289 3310 raft_consensus.cc:2258] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:58.221166 3310 ts_tablet_manager.cc:1875] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:58.226233 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:58.230146 3222 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:58.237398 31060 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:58.268784 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:58.268988 30662 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:58.269982 28603 catalog_manager.cc:5484] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:58.279148 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet e821b93aa9b84ff3a968860a5b33b918 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59004
I20240501 14:02:58.307672 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:58.318245 3310 ts_tablet_manager.cc:1888] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:58.318444 3310 log.cc:1196] T 6b6a54ac32694640bffdf1c7f3b035cc P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/6b6a54ac32694640bffdf1c7f3b035cc
I20240501 14:02:58.319468 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet 6b6a54ac32694640bffdf1c7f3b035cc (table rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]) successfully deleted
I20240501 14:02:58.341487 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:58.343955 3327 ts_tablet_manager.cc:902] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:02:58.346081 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:58.346176 28744 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:02:58.356119 3328 tablet_replica.cc:330] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:02:58.356374 3328 raft_consensus.cc:2229] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:58.356586 3328 raft_consensus.cc:2258] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:58.357386 3328 ts_tablet_manager.cc:1875] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:58.390978 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:58.408421 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:58.403478 3328 ts_tablet_manager.cc:1888] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:58.408840 3328 log.cc:1196] T e821b93aa9b84ff3a968860a5b33b918 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/e821b93aa9b84ff3a968860a5b33b918
I20240501 14:02:58.409758 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet e821b93aa9b84ff3a968860a5b33b918 (table rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]) successfully deleted
I20240501 14:02:58.410447 3327 tablet_copy_client.cc:308] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:02:58.411347 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 6aaf52ee71b44511b7b8909711ab0a42 from peer 3a3972b170c54344a4ba61f176f837ba ({username='slave'} at 127.16.36.132:53415)
I20240501 14:02:58.411501 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 6aaf52ee71b44511b7b8909711ab0a42 from peer 3a3972b170c54344a4ba61f176f837ba at {username='slave'} at 127.16.36.132:53415: session id = 3a3972b170c54344a4ba61f176f837ba-6aaf52ee71b44511b7b8909711ab0a42
I20240501 14:02:58.414911 28765 tablet_copy_source_session.cc:205] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:02:58.436576 29136 consensus_queue.cc:237] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:58.437212 3327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aaf52ee71b44511b7b8909711ab0a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:02:58.439435 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:58.476819 30926 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:58.477468 3300 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:58.480095 3327 tablet_copy_client.cc:783] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 0 data blocks...
I20240501 14:02:58.481091 3327 tablet_copy_client.cc:647] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:02:58.484264 3327 tablet_copy_client.cc:523] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:02:58.488737 28744 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:02:58.489539 3300 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:02:58.512976 3300 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:58.520218 30926 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:58.550032 3327 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:02:58.550460 28603 catalog_manager.cc:5484] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:58.539003 28744 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:58.581492 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 86bfedcdabe44e7ab135674163935786 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59004
I20240501 14:02:58.581974 3328 tablet_replica.cc:330] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:02:58.582244 3328 raft_consensus.cc:2229] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:58.582576 3328 raft_consensus.cc:2258] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:58.583664 3328 ts_tablet_manager.cc:1875] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:58.615991 3328 ts_tablet_manager.cc:1888] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:02:58.616158 3328 log.cc:1196] T 86bfedcdabe44e7ab135674163935786 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/86bfedcdabe44e7ab135674163935786
I20240501 14:02:58.622546 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 86bfedcdabe44e7ab135674163935786 (table rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]) successfully deleted
I20240501 14:02:58.726985 3327 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:02:58.729133 3327 tablet_bootstrap.cc:492] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap complete.
I20240501 14:02:58.729677 3327 ts_tablet_manager.cc:1367] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.180s user 0.071s sys 0.000s
I20240501 14:02:58.731763 3327 raft_consensus.cc:348] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:58.743077 3327 raft_consensus.cc:729] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:02:58.743678 3327 consensus_queue.cc:260] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:02:58.745985 3327 ts_tablet_manager.cc:1398] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:02:58.747613 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 3a3972b170c54344a4ba61f176f837ba-6aaf52ee71b44511b7b8909711ab0a42 received from {username='slave'} at 127.16.36.132:53415
I20240501 14:02:58.747745 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 3a3972b170c54344a4ba61f176f837ba-6aaf52ee71b44511b7b8909711ab0a42 on tablet 6aaf52ee71b44511b7b8909711ab0a42 with peer 3a3972b170c54344a4ba61f176f837ba
I20240501 14:02:58.760152 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:58.768088 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:58.831149 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:58.836095 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:58.848526 29136 raft_consensus.cc:1203] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:02:58.862095 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:58.883869 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:58.922171 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:58.964696 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:02:58.985319 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 48977.8us
I20240501 14:02:59.275950 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:59.312888 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:59.323952 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:59.361147 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:59.371026 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:59.378425 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:59.416157 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:59.453945 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:59.527932 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:59.564735 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:59.572003 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:02:59.586712 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:59.592957 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:02:59.624670 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:02:59.645502 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:02:59.648128 29136 consensus_queue.cc:237] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:59.658839 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:02:59.700594 30928 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:02:59.701411 3300 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:02:59.713760 3300 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:59.718464 30926 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:59.732060 28603 catalog_manager.cc:5484] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:02:59.814148 29136 consensus_queue.cc:237] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:59.814494 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet 86bfedcdabe44e7ab135674163935786 with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:02:59.819278 30926 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:59.820209 3092 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:02:59.825776 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86bfedcdabe44e7ab135674163935786. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:02:59.827080 3092 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:59.834336 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 86bfedcdabe44e7ab135674163935786 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:02:59.876170 3530 tablet_replica.cc:330] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:02:59.876533 3530 raft_consensus.cc:2229] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:02:59.876962 3530 raft_consensus.cc:2258] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:02:59.878229 3530 ts_tablet_manager.cc:1875] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:02:59.878330 28603 catalog_manager.cc:5484] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 6 to 7, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
W20240501 14:02:59.879519 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 53557.1us
I20240501 14:02:59.881712 30926 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:02:59.917690 3530 ts_tablet_manager.cc:1888] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:02:59.922724 3530 log.cc:1196] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/86bfedcdabe44e7ab135674163935786
I20240501 14:02:59.926820 29136 consensus_queue.cc:237] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:59.929104 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet 86bfedcdabe44e7ab135674163935786 (table rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]) successfully deleted
I20240501 14:02:59.960747 30661 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:59.961498 3300 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:02:59.964854 30926 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:02:59.965576 3300 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:00.003214 3532 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:00.004503 30928 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:00.010066 28597 catalog_manager.cc:5484] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:00.015475 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet f516f01e242d4b8789130201f748f3ea with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:00.028983 3563 tablet_replica.cc:330] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:00.039286 3563 raft_consensus.cc:2229] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:00.039814 3563 raft_consensus.cc:2258] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:00.040720 3563 ts_tablet_manager.cc:1875] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:00.045111 30661 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:00.085227 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:00.094322 3563 ts_tablet_manager.cc:1888] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:00.094525 3563 log.cc:1196] T f516f01e242d4b8789130201f748f3ea P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/f516f01e242d4b8789130201f748f3ea
I20240501 14:03:00.099877 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet f516f01e242d4b8789130201f748f3ea (table rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]) successfully deleted
I20240501 14:03:00.132884 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:00.143378 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 65094.7us
I20240501 14:03:00.162426 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:00.196022 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:03:00.200361 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:00.226608 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:00.240353 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:00.241576 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:00.320223 3612 ts_tablet_manager.cc:902] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:03:00.351197 3612 tablet_copy_client.cc:308] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:03:00.352382 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 86bfedcdabe44e7ab135674163935786 from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:48133)
I20240501 14:03:00.352551 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 86bfedcdabe44e7ab135674163935786 from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:48133: session id = 26acba62745c48e3ba8180748d55b211-86bfedcdabe44e7ab135674163935786
I20240501 14:03:00.356537 29158 tablet_copy_source_session.cc:205] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:00.358408 3612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bfedcdabe44e7ab135674163935786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:00.378913 28873 consensus_queue.cc:237] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:00.410316 3612 tablet_copy_client.cc:783] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:00.411128 3612 tablet_copy_client.cc:647] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:00.416671 3612 tablet_copy_client.cc:523] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:00.419075 30793 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:00.420908 3222 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:00.421149 30661 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:00.420310 29007 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:00.427783 3222 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:00.428877 3222 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:00.474771 3612 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
I20240501 14:03:00.485105 3552 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:00.489260 30795 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:00.506105 28597 catalog_manager.cc:5484] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:00.576879 29006 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:00.587724 30661 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:00.595764 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet bb712628c7764a138f5425d14b724d9e with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:03:00.613157 3654 tablet_replica.cc:330] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:03:00.635867 3654 raft_consensus.cc:2229] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:00.637355 3654 raft_consensus.cc:2258] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:00.641973 3654 ts_tablet_manager.cc:1875] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:00.684906 3654 ts_tablet_manager.cc:1888] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:00.685360 3654 log.cc:1196] T bb712628c7764a138f5425d14b724d9e P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/bb712628c7764a138f5425d14b724d9e
I20240501 14:03:00.687017 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet bb712628c7764a138f5425d14b724d9e (table rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]) successfully deleted
I20240501 14:03:00.699185 3612 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:00.699726 3612 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:03:00.700119 3612 ts_tablet_manager.cc:1367] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.226s user 0.073s sys 0.001s
I20240501 14:03:00.701066 3612 raft_consensus.cc:348] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:00.701326 3612 raft_consensus.cc:729] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:03:00.707054 3612 consensus_queue.cc:260] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:00.709007 3612 ts_tablet_manager.cc:1398] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:03:00.709822 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-86bfedcdabe44e7ab135674163935786 received from {username='slave'} at 127.16.36.133:48133
I20240501 14:03:00.709954 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 26acba62745c48e3ba8180748d55b211-86bfedcdabe44e7ab135674163935786 on tablet 86bfedcdabe44e7ab135674163935786 with peer 26acba62745c48e3ba8180748d55b211
W20240501 14:03:00.733304 28590 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 46303.9us
I20240501 14:03:00.775588 30661 raft_consensus.cc:1203] T 86bfedcdabe44e7ab135674163935786 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:03:00.854048 29136 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:00.858698 30926 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:00.859431 3532 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:00.868065 3300 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:00.877480 28595 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:00.891949 29136 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
W20240501 14:03:00.899700 29072 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba -> Peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): Couldn't send request to peer 7e7bc120bd74479ab378f779778fd1e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60b3db79ff754e8bbe9e5865f7104312. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:00.900877 30928 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:00.901743 3532 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:00.903291 30928 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:00.960819 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:00.979027 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b3db79ff754e8bbe9e5865f7104312 with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:03:00.979403 3654 tablet_replica.cc:330] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:03:00.979621 3654 raft_consensus.cc:2229] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:00.979969 3654 raft_consensus.cc:2258] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:00.980741 3654 ts_tablet_manager.cc:1875] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:00.989543 3654 ts_tablet_manager.cc:1888] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:00.989804 3654 log.cc:1196] T 60b3db79ff754e8bbe9e5865f7104312 P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/60b3db79ff754e8bbe9e5865f7104312
I20240501 14:03:00.991281 3532 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:03:00.992835 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet 60b3db79ff754e8bbe9e5865f7104312 (table rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]) successfully deleted
I20240501 14:03:00.995951 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 60b3db79ff754e8bbe9e5865f7104312 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:00.996805 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:00.997828 28595 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:03:01.011225 30926 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:03:01.030342 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:01.052537 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:03:01.112622 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:01.151226 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:01.165279 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:01.176208 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:01.480257 3799 ts_tablet_manager.cc:902] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:03:01.510466 3799 tablet_copy_client.cc:308] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:03:01.511652 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 60b3db79ff754e8bbe9e5865f7104312 from peer 7e7bc120bd74479ab378f779778fd1e8 ({username='slave'} at 127.16.36.131:37889)
I20240501 14:03:01.511834 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 60b3db79ff754e8bbe9e5865f7104312 from peer 7e7bc120bd74479ab378f779778fd1e8 at {username='slave'} at 127.16.36.131:37889: session id = 7e7bc120bd74479ab378f779778fd1e8-60b3db79ff754e8bbe9e5865f7104312
I20240501 14:03:01.515528 29158 tablet_copy_source_session.cc:205] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:01.517966 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b3db79ff754e8bbe9e5865f7104312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:01.567080 3799 tablet_copy_client.cc:783] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:01.567477 3799 tablet_copy_client.cc:647] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:01.575852 3799 tablet_copy_client.cc:523] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:01.580279 3799 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:03:01.613157 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:03:01.613487 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:01.671550 29136 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:01.701244 30795 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:01.701872 3720 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:01.702764 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:01.711757 28873 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:01.714607 3720 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:01.715880 3720 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:01.720494 3799 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:01.720968 3799 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap complete.
I20240501 14:03:01.721221 3799 ts_tablet_manager.cc:1367] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.141s user 0.073s sys 0.014s
I20240501 14:03:01.721980 3799 raft_consensus.cc:348] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:03:01.722323 3799 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:03:01.722656 3799 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:03:01.724175 30793 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:01.729522 28603 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:01.733016 3799 ts_tablet_manager.cc:1398] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:03:01.734990 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 7e7bc120bd74479ab378f779778fd1e8-60b3db79ff754e8bbe9e5865f7104312 received from {username='slave'} at 127.16.36.131:37889
I20240501 14:03:01.735126 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 7e7bc120bd74479ab378f779778fd1e8-60b3db79ff754e8bbe9e5865f7104312 on tablet 60b3db79ff754e8bbe9e5865f7104312 with peer 7e7bc120bd74479ab378f779778fd1e8
I20240501 14:03:01.751034 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:01.780169 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 2020fe37059a4e74bb45190398e4e177 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:01.780995 3864 tablet_replica.cc:330] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:01.781332 3864 raft_consensus.cc:2229] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:01.781635 3864 raft_consensus.cc:2258] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:01.783077 3864 ts_tablet_manager.cc:1875] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:01.792986 3864 ts_tablet_manager.cc:1888] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:03:01.797740 3864 log.cc:1196] T 2020fe37059a4e74bb45190398e4e177 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/2020fe37059a4e74bb45190398e4e177
I20240501 14:03:01.799408 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 2020fe37059a4e74bb45190398e4e177 (table rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]) successfully deleted
I20240501 14:03:01.885349 29006 raft_consensus.cc:1203] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:03:01.934128 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:03:01.942375 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:01.945745 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:01.944885 28873 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:01.960043 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.012068 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.030179 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:02.041625 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.094307 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:02.119422 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:02.160029 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:02.162226 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.227257 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:02.243417 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:02.279650 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:02.291263 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.310184 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.353753 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:03:02.354346 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:02.381902 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.437079 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:02.525974 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.539543 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:02.556810 28986 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:03:02.559906 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:02.605131 31041 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.660729 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:02.661165 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:02.673223 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:03.100389 29006 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8"
dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32834
I20240501 14:03:03.100713 29006 raft_consensus.cc:595] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:03.104866 29006 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:03.138168 30795 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:03.139134 3585 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:03.160149 3585 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:03.166602 30793 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:03.167449 29006 consensus_queue.cc:237] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:03.194319 28596 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 5 to 6, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:03.245604 4008 raft_consensus.cc:979] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: : Instructing follower 826ccf8358b6453d88bcb132290b82a5 to start an election
I20240501 14:03:03.245851 4008 raft_consensus.cc:1067] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Signalling peer 826ccf8358b6453d88bcb132290b82a5 to start an election
W20240501 14:03:03.262851 28941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 73200.5us
I20240501 14:03:03.263136 29136 raft_consensus.cc:1261] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:03.263898 31059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8"
dest_uuid: "826ccf8358b6453d88bcb132290b82a5"
from {username='slave'} at 127.16.36.131:37693
I20240501 14:03:03.264359 31059 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:03.264616 31059 raft_consensus.cc:3037] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:03.269009 3585 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:03.269414 31059 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:03.270932 31060 raft_consensus.cc:1261] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:03.272753 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "826ccf8358b6453d88bcb132290b82a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:03:03.272989 28874 raft_consensus.cc:3037] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:03.273916 29006 consensus_queue.cc:237] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:03.275323 4005 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:03.276327 28874 raft_consensus.cc:2454] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826ccf8358b6453d88bcb132290b82a5 in term 2.
I20240501 14:03:03.286553 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet efe7fed8e5d44e8a80ffbdc27619a74d with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46536
I20240501 14:03:03.286741 29007 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:03.293454 4005 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:03.309312 4020 tablet_replica.cc:330] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:03:03.310505 31059 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [CANDIDATE]: Term 2 election: Requested vote from peers cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:03:03.311451 4020 raft_consensus.cc:2229] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:03.311772 4020 raft_consensus.cc:2258] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:03.316311 4020 ts_tablet_manager.cc:1875] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:03.326525 28874 raft_consensus.cc:1226] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Rejecting Update request from peer 7e7bc120bd74479ab378f779778fd1e8 for earlier term 1. Current term is 2. Ops: []
I20240501 14:03:03.326926 4020 ts_tablet_manager.cc:1888] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:03.327181 4020 log.cc:1196] T efe7fed8e5d44e8a80ffbdc27619a74d P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/efe7fed8e5d44e8a80ffbdc27619a74d
I20240501 14:03:03.327574 29136 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:03.329407 28597 catalog_manager.cc:5484] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:03.330762 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "826ccf8358b6453d88bcb132290b82a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:03:03.331030 29007 raft_consensus.cc:3032] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:03.331130 29007 raft_consensus.cc:729] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:03:03.331583 29007 consensus_queue.cc:260] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:03.332106 29007 raft_consensus.cc:3037] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:03.332177 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet efe7fed8e5d44e8a80ffbdc27619a74d (table rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]) successfully deleted
I20240501 14:03:03.333653 28743 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:03.333735 30661 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:03.334425 4008 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:03.335175 4008 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:03.338475 29007 raft_consensus.cc:2454] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826ccf8358b6453d88bcb132290b82a5 in term 2.
I20240501 14:03:03.338817 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 111ba172d7e44054872c8fc8cac8ba88 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:03.348364 30795 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:03.351337 4027 tablet_replica.cc:330] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:03.353094 4027 raft_consensus.cc:2229] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:03.355229 4027 raft_consensus.cc:2258] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:03.357309 4027 ts_tablet_manager.cc:1875] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:03.366179 30995 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, 826ccf8358b6453d88bcb132290b82a5, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:03:03.368237 4008 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:03:03.369102 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efe7fed8e5d44e8a80ffbdc27619a74d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:03.378437 31059 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:03.380694 4005 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:03.380357 31060 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:03.382608 4008 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:03.389137 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet efe7fed8e5d44e8a80ffbdc27619a74d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:03.391569 3585 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:03.396282 28745 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:03.404592 28597 catalog_manager.cc:5484] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:03.405543 28596 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:03.410449 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet fb04ec77da194832aa63d69aba307de3 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:03.410979 4032 raft_consensus.cc:2781] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:03.411036 30795 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
W20240501 14:03:03.421047 30995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 54939.9us
I20240501 14:03:03.457942 30928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "826ccf8358b6453d88bcb132290b82a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a0e901103604414c829b40edf9f24c28"
I20240501 14:03:03.458527 30928 raft_consensus.cc:3037] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:03.464655 30928 raft_consensus.cc:2454] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 826ccf8358b6453d88bcb132290b82a5 in term 2.
I20240501 14:03:03.467496 30661 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:03.472697 4032 raft_consensus.cc:686] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 2 LEADER]: Becoming Leader. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Running, Role: LEADER
I20240501 14:03:03.473742 4032 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:03.479873 31059 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:03.496850 4027 ts_tablet_manager.cc:1888] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:03:03.497170 4027 log.cc:1196] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/111ba172d7e44054872c8fc8cac8ba88
I20240501 14:03:03.520499 4027 tablet_replica.cc:330] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:03.520581 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 111ba172d7e44054872c8fc8cac8ba88 (table rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]) successfully deleted
I20240501 14:03:03.524892 4027 raft_consensus.cc:2229] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:03.526124 4027 raft_consensus.cc:2258] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:03.527730 4027 ts_tablet_manager.cc:1875] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:03.529934 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:03.531143 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:03.571523 4027 ts_tablet_manager.cc:1888] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:03.578069 4027 log.cc:1196] T fb04ec77da194832aa63d69aba307de3 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb04ec77da194832aa63d69aba307de3
I20240501 14:03:03.582566 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet fb04ec77da194832aa63d69aba307de3 (table rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]) successfully deleted
I20240501 14:03:03.591351 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:03.594830 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:03.636938 28596 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 reported cstate change: term changed from 1 to 2, leader changed from 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) to 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136). New cstate: current_term: 2 leader_uuid: "826ccf8358b6453d88bcb132290b82a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:03.638615 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:03.664644 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:03.693104 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:03.713282 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:03.812520 4099 ts_tablet_manager.cc:902] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:03:03.866333 4099 tablet_copy_client.cc:308] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:03:03.867730 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet efe7fed8e5d44e8a80ffbdc27619a74d from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43429)
I20240501 14:03:03.867899 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet efe7fed8e5d44e8a80ffbdc27619a74d from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43429: session id = 826ccf8358b6453d88bcb132290b82a5-efe7fed8e5d44e8a80ffbdc27619a74d
I20240501 14:03:03.872058 29027 tablet_copy_source_session.cc:205] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:03.880822 4099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe7fed8e5d44e8a80ffbdc27619a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:03.914763 4099 tablet_copy_client.cc:783] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:03.916960 4099 tablet_copy_client.cc:647] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:03.920154 4099 tablet_copy_client.cc:523] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:03.945678 28874 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Refusing update from remote peer 826ccf8358b6453d88bcb132290b82a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:03.949538 4038 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:03.951834 30926 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 826ccf8358b6453d88bcb132290b82a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:03.956429 4032 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:03.964694 29007 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer 826ccf8358b6453d88bcb132290b82a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:03.971017 4032 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:04.074823 4099 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:03:04.177490 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:04.218888 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:04.242869 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:04.246052 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:04.274590 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:04.290575 30775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:04.305866 29007 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87"
dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32876
I20240501 14:03:04.306108 29007 raft_consensus.cc:595] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:04.309916 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:04.363618 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:04.387181 4008 raft_consensus.cc:979] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: : Instructing follower 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:04.387724 4005 raft_consensus.cc:1067] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Signalling peer 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:04.389055 30795 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87"
dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf"
from {username='slave'} at 127.16.36.131:45889
I20240501 14:03:04.389394 30795 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:04.389611 30795 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:04.393613 30795 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:04.398555 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:03:04.398901 29007 raft_consensus.cc:3032] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:04.398998 29007 raft_consensus.cc:729] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:03:04.399281 29007 consensus_queue.cc:260] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:04.399571 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:03:04.399807 29007 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:04.399854 29136 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:04.403179 29007 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:04.410173 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:04.421115 30728 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8, 8fc5d8222c164a679e7a99966da1eaaf; no voters:
I20240501 14:03:04.421628 2743 raft_consensus.cc:2781] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:04.427567 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:04.433460 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:04.441113 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:04.444372 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:04.464573 30795 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:03:04.481047 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:04.563340 2743 raft_consensus.cc:686] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Becoming Leader. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Running, Role: LEADER
I20240501 14:03:04.564350 2743 consensus_queue.cc:237] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:04.593034 30661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "26acba62745c48e3ba8180748d55b211"
I20240501 14:03:04.593556 30661 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:04.597942 30661 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
W20240501 14:03:04.622014 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 212316us
W20240501 14:03:04.684221 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 45917.7us
I20240501 14:03:04.687602 4099 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:04.688297 4099 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:03:04.688633 4099 ts_tablet_manager.cc:1367] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.614s user 0.063s sys 0.007s
I20240501 14:03:04.689862 4099 raft_consensus.cc:348] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:04.690203 4099 raft_consensus.cc:729] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:03:04.697657 4099 consensus_queue.cc:260] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:04.698462 4099 ts_tablet_manager.cc:1398] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:03:04.699887 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-efe7fed8e5d44e8a80ffbdc27619a74d received from {username='slave'} at 127.16.36.136:43429
I20240501 14:03:04.700021 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-efe7fed8e5d44e8a80ffbdc27619a74d on tablet efe7fed8e5d44e8a80ffbdc27619a74d with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:04.703513 28596 catalog_manager.cc:5484] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: term changed from 1 to 2, leader changed from 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) to 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134). New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:04.705114 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:04.734947 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:04.790881 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a0e901103604414c829b40edf9f24c28"
I20240501 14:03:04.791182 30926 raft_consensus.cc:3037] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:04.794180 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:04.804661 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:04.948413 31059 raft_consensus.cc:1203] T efe7fed8e5d44e8a80ffbdc27619a74d P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:03:04.962008 30661 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:04.962783 2743 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:04.994835 29136 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:04.996639 2743 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:05.005221 30926 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:05.006040 2743 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:05.063781 29007 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:05.064901 4199 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:05.116318 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.130957 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:05.201439 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:05.335919 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:05.369239 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:05.382994 29117 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:05.424327 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:05.426384 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.443087 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:05.486809 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:05.489069 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:05.490365 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:05.506693 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:05.525532 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:05.539203 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:03:05.549973 28941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 95777us
I20240501 14:03:05.605321 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:05.782171 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 93075.4us
W20240501 14:03:05.790030 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 102744us
I20240501 14:03:05.955124 29136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db"
dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59758
I20240501 14:03:05.955438 29136 raft_consensus.cc:595] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:05.980636 4322 raft_consensus.cc:979] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: : Instructing follower cd3c04ebb6c84b15a1b3adc748737094 to start an election
I20240501 14:03:05.980851 4322 raft_consensus.cc:1067] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Signalling peer cd3c04ebb6c84b15a1b3adc748737094 to start an election
I20240501 14:03:05.981922 28874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db"
dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
from {username='slave'} at 127.16.36.132:52759
I20240501 14:03:05.982200 28874 raft_consensus.cc:482] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:05.982318 28874 raft_consensus.cc:3037] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:05.988241 28874 raft_consensus.cc:504] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:05.989797 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:03:05.990118 29136 raft_consensus.cc:3032] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:05.990212 29136 raft_consensus.cc:729] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:03:05.990556 29136 consensus_queue.cc:260] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:05.991094 29136 raft_consensus.cc:3037] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:05.990495 28743 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:05.998503 29136 raft_consensus.cc:2454] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 2.
I20240501 14:03:06.000712 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a0e901103604414c829b40edf9f24c28"
I20240501 14:03:06.001001 30926 raft_consensus.cc:3037] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:06.004453 30926 raft_consensus.cc:2454] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 2.
I20240501 14:03:06.004616 31059 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.005120 28811 leader_election.cc:302] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:03:06.005390 4243 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.005540 28874 leader_election.cc:288] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 2 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:03:06.008901 4076 raft_consensus.cc:2781] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:06.010803 29007 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.011559 4243 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.013864 4076 raft_consensus.cc:686] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 2 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:03:06.014312 4076 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:06.031545 28595 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 1 to 2, leader changed from 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 2 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:06.032891 4243 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:06.039367 30661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de161b10c84f4bcc85f8cf7b199f99db" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "26acba62745c48e3ba8180748d55b211"
I20240501 14:03:06.039714 30661 raft_consensus.cc:3037] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:06.040146 31059 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:06.045218 30661 raft_consensus.cc:2454] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 2.
I20240501 14:03:06.066406 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:06.067452 28598 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:06.086859 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b473e10a8ad4bd58bb44aabd9656e08 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:06.098673 4385 tablet_replica.cc:330] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:06.099090 4385 raft_consensus.cc:2229] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:06.099452 4385 raft_consensus.cc:2258] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:06.100441 4385 ts_tablet_manager.cc:1875] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:06.103952 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:06.136626 29007 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:06.144727 29117 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:06.173954 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:06.176496 4385 ts_tablet_manager.cc:1888] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:03:06.184859 4385 log.cc:1196] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b473e10a8ad4bd58bb44aabd9656e08
I20240501 14:03:06.189332 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 9b473e10a8ad4bd58bb44aabd9656e08 (table rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]) successfully deleted
W20240501 14:03:06.202032 30730 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 69365us
I20240501 14:03:06.206733 31059 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:06.249850 30926 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 826ccf8358b6453d88bcb132290b82a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:06.250514 28874 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Refusing update from remote peer 826ccf8358b6453d88bcb132290b82a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:06.251477 4038 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:06.278259 4038 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:06.280104 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:06.280221 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:06.297713 28874 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:06.296228 4038 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:06.344311 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:06.347388 28601 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 reported cstate change: config changed from index 6 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "826ccf8358b6453d88bcb132290b82a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:06.346956 30926 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:06.352555 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b9db597e654ed48b679abf58b8a2c8 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:06.377933 4426 tablet_replica.cc:330] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:06.378347 4426 raft_consensus.cc:2229] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:06.381338 4426 raft_consensus.cc:2258] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:06.385538 4426 ts_tablet_manager.cc:1875] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:06.403231 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:06.406598 4426 ts_tablet_manager.cc:1888] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 14:03:06.406869 4426 log.cc:1196] T e8b9db597e654ed48b679abf58b8a2c8 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/e8b9db597e654ed48b679abf58b8a2c8
I20240501 14:03:06.407769 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet e8b9db597e654ed48b679abf58b8a2c8 (table rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]) successfully deleted
I20240501 14:03:06.434692 29007 consensus_queue.cc:237] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:06.442404 30794 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.445295 4008 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.451042 29136 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:06.451819 4294 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:06.458762 29138 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:06.459595 4363 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:06.475080 4294 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:06.476900 30795 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:06.499501 28601 catalog_manager.cc:5484] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:06.506975 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8c435795a04b1b90b6e2ff914676c8 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:06.507334 4385 tablet_replica.cc:330] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:06.507519 4385 raft_consensus.cc:2229] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:06.507714 4385 raft_consensus.cc:2258] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:06.508486 4385 ts_tablet_manager.cc:1875] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:06.526429 4385 ts_tablet_manager.cc:1888] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:03:06.527215 4385 log.cc:1196] T 4d8c435795a04b1b90b6e2ff914676c8 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d8c435795a04b1b90b6e2ff914676c8
I20240501 14:03:06.530032 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 4d8c435795a04b1b90b6e2ff914676c8 (table rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]) successfully deleted
I20240501 14:03:06.541824 30661 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:06.542757 4363 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:06.587283 29136 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:06.612496 30926 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:06.613508 4076 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:06.899278 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:06.978694 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:06.993331 29117 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.055966 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:07.120090 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:07.131650 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.156190 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:07.174153 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.184211 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:07.184188 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.232571 29117 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.251724 30794 consensus_queue.cc:237] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:07.272753 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.299552 30661 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:07.300287 2743 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:07.302031 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:07.306378 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:07.306928 29136 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:07.307602 2743 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:07.320564 30928 raft_consensus.cc:1261] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:07.321367 2743 consensus_queue.cc:1035] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:07.358446 4588 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:07.381992 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:07.420750 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:07.436769 30928 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:07.462370 29136 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:07.483515 30661 raft_consensus.cc:2932] T 1fdedd39e12a49659d989f37b3287d87 P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:07.494335 28597 catalog_manager.cc:5484] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:07.627193 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:07.642242 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdedd39e12a49659d989f37b3287d87 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:07.675733 4632 tablet_replica.cc:330] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:07.676064 4632 raft_consensus.cc:2229] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:07.676301 4632 raft_consensus.cc:2258] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:07.677289 4632 ts_tablet_manager.cc:1875] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:07.687744 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:07.696892 4632 ts_tablet_manager.cc:1888] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:03:07.697162 4632 log.cc:1196] T 1fdedd39e12a49659d989f37b3287d87 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/1fdedd39e12a49659d989f37b3287d87
I20240501 14:03:07.698168 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 1fdedd39e12a49659d989f37b3287d87 (table rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]) successfully deleted
I20240501 14:03:07.717269 29117 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.776014 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.779942 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:07.812371 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:07.832537 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:07.833887 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.872956 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:07.932852 29117 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.947450 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:03:07.958177 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:07.958570 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:07.966046 30775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:03:08.015343 28680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 81831us
I20240501 14:03:08.057731 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:08.108376 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:08.462280 28743 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:08.492667 31059 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.493594 4587 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.494963 28874 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663"
dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60764
I20240501 14:03:08.495220 28874 raft_consensus.cc:595] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:08.508092 30928 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.509001 4587 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.531890 29007 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:08.535084 4587 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:08.607591 4483 raft_consensus.cc:979] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: : Instructing follower 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:08.607829 4483 raft_consensus.cc:1067] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Signalling peer 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:08.609115 30794 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663"
dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf"
from {username='slave'} at 127.16.36.130:55461
I20240501 14:03:08.609401 30794 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:08.609557 30794 raft_consensus.cc:3037] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:08.615322 30794 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:08.616469 30794 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:03:08.617421 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:03:08.617692 29136 raft_consensus.cc:3037] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:08.621632 29136 raft_consensus.cc:2454] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:08.627242 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:03:08.630261 28874 raft_consensus.cc:3032] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:08.630389 28874 raft_consensus.cc:729] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:03:08.630815 28874 consensus_queue.cc:260] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:08.631407 28874 raft_consensus.cc:3037] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:08.631461 31059 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:08.635273 30728 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 8fc5d8222c164a679e7a99966da1eaaf; no voters:
I20240501 14:03:08.635748 2743 raft_consensus.cc:2781] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:08.638154 28598 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 reported cstate change: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:08.638247 4587 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:08.645107 28874 raft_consensus.cc:2454] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:08.645268 29007 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:08.662896 28874 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:08.666738 30928 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:08.677551 2743 raft_consensus.cc:686] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Becoming Leader. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Running, Role: LEADER
I20240501 14:03:08.681705 2743 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:08.692811 30661 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:08.694900 4725 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:08.709841 30662 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:08.714824 4726 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:08.724160 28597 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: term changed from 1 to 2, leader changed from cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) to 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134). New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:08.752749 28595 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P 26acba62745c48e3ba8180748d55b211 reported cstate change: config changed from index 6 to 8, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 2 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:08.760530 30926 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:08.761500 4483 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:08.772321 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:08.773471 30928 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:08.786837 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:08.791389 29117 tablet_service.cc:1514] Processing DeleteTablet for tablet de161b10c84f4bcc85f8cf7b199f99db with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:08.796276 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet e0bf08cbb6d347d783b1fa7762c802df with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:08.818598 4772 tablet_replica.cc:330] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:08.821075 4772 raft_consensus.cc:2229] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:08.822394 4772 raft_consensus.cc:2258] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:08.823534 4772 ts_tablet_manager.cc:1875] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:08.832500 29117 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:08.833817 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:08.852803 4772 ts_tablet_manager.cc:1888] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 14:03:08.853176 4772 log.cc:1196] T de161b10c84f4bcc85f8cf7b199f99db P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/de161b10c84f4bcc85f8cf7b199f99db
I20240501 14:03:08.854342 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet de161b10c84f4bcc85f8cf7b199f99db (table rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]) successfully deleted
I20240501 14:03:08.894394 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:08.920938 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:08.975165 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:08.992713 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:09.003489 4772 tablet_replica.cc:330] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:09.003751 4772 raft_consensus.cc:2229] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:09.004005 4772 raft_consensus.cc:2258] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:09.005039 4772 ts_tablet_manager.cc:1875] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:09.013612 4772 ts_tablet_manager.cc:1888] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:09.015548 4772 log.cc:1196] T e0bf08cbb6d347d783b1fa7762c802df P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/e0bf08cbb6d347d783b1fa7762c802df
I20240501 14:03:09.017822 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet e0bf08cbb6d347d783b1fa7762c802df (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
I20240501 14:03:09.151813 29136 consensus_queue.cc:237] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:09.156102 31059 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.157284 4574 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.160951 30662 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.161927 30794 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.162859 29007 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:09.170579 4314 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.176440 4574 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.177292 4574 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:09.192188 29007 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:09.192093 4314 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:09.200764 30794 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:09.205850 28874 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:03:09.206605 2743 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:09.212843 31059 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:09.219594 28598 catalog_manager.cc:5484] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:09.208581 30662 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:09.238392 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet f4402de0b48041c383568feb9365147e with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:09.244433 29136 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:03:09.245245 4734 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:09.264467 4834 tablet_replica.cc:330] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:09.265379 4834 raft_consensus.cc:2229] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:09.266119 4834 raft_consensus.cc:2258] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:09.267608 4834 ts_tablet_manager.cc:1875] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:09.311426 4834 ts_tablet_manager.cc:1888] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:09.312363 4834 log.cc:1196] T f4402de0b48041c383568feb9365147e P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/f4402de0b48041c383568feb9365147e
I20240501 14:03:09.313817 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet f4402de0b48041c383568feb9365147e (table rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]) successfully deleted
I20240501 14:03:09.587007 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:09.603216 29118 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:09.634565 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:09.653201 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:09.654325 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:09.682452 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:09.712579 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:09.718395 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:09.721112 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:09.775440 29118 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:09.781014 28874 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:09.788872 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:09.790959 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:09.839917 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:09.860718 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:03:09.873028 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 47542.4us
I20240501 14:03:09.875588 30928 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:09.876252 4918 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:09.888808 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:09.909950 4918 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:09.927165 30928 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:09.953356 28598 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:09.967085 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:10.033051 28874 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:03:10.036104 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6feda9b53e484e7ea40a5890b7d51ac4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:10.036322 30926 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:03:10.037935 4725 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:10.041342 4725 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:10.045728 30926 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:10.053381 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6feda9b53e484e7ea40a5890b7d51ac4 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:10.057335 28601 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 8 to 9, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:10.107882 31041 tablet_service.cc:1514] Processing DeleteTablet for tablet 6feda9b53e484e7ea40a5890b7d51ac4 with delete_type TABLET_DATA_TOMBSTONED (TS 826ccf8358b6453d88bcb132290b82a5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57628
I20240501 14:03:10.124544 4982 tablet_replica.cc:330] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: stopping tablet replica
I20240501 14:03:10.125134 4982 raft_consensus.cc:2229] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:10.125650 4982 raft_consensus.cc:2258] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:10.127298 4982 ts_tablet_manager.cc:1875] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:10.137526 4982 ts_tablet_manager.cc:1888] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:03:10.137709 4982 log.cc:1196] T 6feda9b53e484e7ea40a5890b7d51ac4 P 826ccf8358b6453d88bcb132290b82a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/wals/6feda9b53e484e7ea40a5890b7d51ac4
I20240501 14:03:10.138669 28589 catalog_manager.cc:4830] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): tablet 6feda9b53e484e7ea40a5890b7d51ac4 (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
I20240501 14:03:10.412813 30794 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:10.439988 29136 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:10.440834 4734 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.471074 5057 ts_tablet_manager.cc:902] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:03:10.492292 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:10.493861 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:10.519263 5063 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:10.526203 29118 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:10.535135 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:10.555056 5057 tablet_copy_client.cc:308] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:03:10.556133 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:52893)
I20240501 14:03:10.556346 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:52893: session id = 8fc5d8222c164a679e7a99966da1eaaf-6feda9b53e484e7ea40a5890b7d51ac4
I20240501 14:03:10.560146 29136 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:10.565018 28896 tablet_copy_source_session.cc:205] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:10.566793 28601 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 5 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:10.569870 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6feda9b53e484e7ea40a5890b7d51ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:10.583732 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:10.606328 5057 tablet_copy_client.cc:783] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:10.606971 5057 tablet_copy_client.cc:647] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:10.609990 5057 tablet_copy_client.cc:523] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:10.614802 5057 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap starting.
W20240501 14:03:10.624533 28940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:37325, user_credentials={real_user=slave}} blocked reactor thread for 54705.5us
I20240501 14:03:10.625351 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:10.657368 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:10.675648 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:10.689478 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:10.746955 5104 tablet_replica.cc:330] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:10.747292 5104 raft_consensus.cc:2229] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:10.747642 5104 raft_consensus.cc:2258] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:10.752777 28745 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:10.756767 29007 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.757536 5104 ts_tablet_manager.cc:1875] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:10.757826 4587 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.761812 30793 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:10.762351 30926 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:10.763202 4587 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:10.765759 29136 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:10.766734 5063 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:10.770009 5104 ts_tablet_manager.cc:1888] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:03:10.770139 5104 log.cc:1196] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/63b7f2a8d8a84faf86c4de16b3cc6663
I20240501 14:03:10.771062 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 63b7f2a8d8a84faf86c4de16b3cc6663 (table rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]) successfully deleted
I20240501 14:03:10.770982 5063 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:10.775853 29138 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:10.778263 4587 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:10.781884 29007 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:10.786491 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:10.787983 28601 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:10.793385 28599 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:10.799015 31041 tablet_service.cc:1514] Processing DeleteTablet for tablet e0bf08cbb6d347d783b1fa7762c802df with delete_type TABLET_DATA_TOMBSTONED (TS 826ccf8358b6453d88bcb132290b82a5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57628
I20240501 14:03:10.821455 5116 tablet_replica.cc:330] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: stopping tablet replica
I20240501 14:03:10.821852 5116 raft_consensus.cc:2229] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:10.822129 5116 raft_consensus.cc:2258] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:10.823522 5116 ts_tablet_manager.cc:1875] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:10.825829 30926 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
W20240501 14:03:10.849454 28679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 58077.4us
I20240501 14:03:10.876281 5116 ts_tablet_manager.cc:1888] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:10.876441 5116 log.cc:1196] T e0bf08cbb6d347d783b1fa7762c802df P 826ccf8358b6453d88bcb132290b82a5: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-7/wal/wals/e0bf08cbb6d347d783b1fa7762c802df
I20240501 14:03:10.877315 28589 catalog_manager.cc:4830] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): tablet e0bf08cbb6d347d783b1fa7762c802df (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
I20240501 14:03:10.906566 5057 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:10.907248 5057 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: Bootstrap complete.
I20240501 14:03:10.907620 5057 ts_tablet_manager.cc:1367] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent bootstrapping tablet: real 0.293s user 0.069s sys 0.017s
I20240501 14:03:10.928112 5057 raft_consensus.cc:348] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:03:10.953357 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63b7f2a8d8a84faf86c4de16b3cc6663. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:10.953852 5057 raft_consensus.cc:729] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Initialized, Role: FOLLOWER
I20240501 14:03:10.954452 5057 consensus_queue.cc:260] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:10.956115 5057 ts_tablet_manager.cc:1398] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20240501 14:03:10.957373 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-6feda9b53e484e7ea40a5890b7d51ac4 received from {username='slave'} at 127.16.36.134:52893
I20240501 14:03:10.957561 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 8fc5d8222c164a679e7a99966da1eaaf-6feda9b53e484e7ea40a5890b7d51ac4 on tablet 6feda9b53e484e7ea40a5890b7d51ac4 with peer 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:10.996078 30793 raft_consensus.cc:1203] T 6feda9b53e484e7ea40a5890b7d51ac4 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:03:11.106650 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.148224 29118 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:11.156260 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:11.211863 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:11.235366 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:11.258805 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:11.269032 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:11.275652 5190 ts_tablet_manager.cc:902] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:11.310184 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.363940 5190 tablet_copy_client.cc:308] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.134:44241
I20240501 14:03:11.407765 30816 tablet_copy_service.cc:140] P 8fc5d8222c164a679e7a99966da1eaaf: Received BeginTabletCopySession request for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:43619)
I20240501 14:03:11.407987 30816 tablet_copy_service.cc:161] P 8fc5d8222c164a679e7a99966da1eaaf: Beginning new tablet copy session on tablet 63b7f2a8d8a84faf86c4de16b3cc6663 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:43619: session id = 826ccf8358b6453d88bcb132290b82a5-63b7f2a8d8a84faf86c4de16b3cc6663
I20240501 14:03:11.419581 30816 tablet_copy_source_session.cc:205] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:11.421638 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7f2a8d8a84faf86c4de16b3cc6663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:11.442476 5190 tablet_copy_client.cc:783] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:11.443099 5190 tablet_copy_client.cc:647] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:11.446936 5190 tablet_copy_client.cc:523] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:11.458987 5190 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:03:11.480099 28874 consensus_queue.cc:237] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:11.504887 30926 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.506170 30793 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.507236 4918 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.507890 4918 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.527313 28745 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:11.530078 5231 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:11.596988 4918 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:11.612841 30793 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:11.614151 30928 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:11.616956 28745 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:11.624891 28601 catalog_manager.cc:5484] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 5 to 6, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:11.680312 5190 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:11.680882 5190 tablet_bootstrap.cc:492] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:03:11.681341 5190 ts_tablet_manager.cc:1367] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.222s user 0.081s sys 0.047s
I20240501 14:03:11.682444 5190 raft_consensus.cc:348] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:11.682914 5190 raft_consensus.cc:729] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:03:11.683338 5190 consensus_queue.cc:260] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:11.684154 5190 ts_tablet_manager.cc:1398] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:11.687417 30816 tablet_copy_service.cc:342] P 8fc5d8222c164a679e7a99966da1eaaf: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-63b7f2a8d8a84faf86c4de16b3cc6663 received from {username='slave'} at 127.16.36.136:43619
I20240501 14:03:11.687639 30816 tablet_copy_service.cc:433] P 8fc5d8222c164a679e7a99966da1eaaf: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-63b7f2a8d8a84faf86c4de16b3cc6663 on tablet 63b7f2a8d8a84faf86c4de16b3cc6663 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:11.731588 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.768354 28725 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:11.769882 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 2998b60071cd408aa1cd06fb57d76a8f with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:11.772878 29118 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:11.787539 28987 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:11.802915 5276 tablet_replica.cc:330] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:11.807633 5276 raft_consensus.cc:2229] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:11.808233 5276 raft_consensus.cc:2258] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:11.810096 5276 ts_tablet_manager.cc:1875] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:11.821040 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:11.852731 28745 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df"
dest_uuid: "343a85ba3b514663af5320cd14a9e144"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51848
I20240501 14:03:11.853096 28745 raft_consensus.cc:595] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:11.861172 31059 raft_consensus.cc:1203] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 14:03:11.861560 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:11.870563 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:11.907721 5276 ts_tablet_manager.cc:1888] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:11.908015 5276 log.cc:1196] T 2998b60071cd408aa1cd06fb57d76a8f P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/2998b60071cd408aa1cd06fb57d76a8f
I20240501 14:03:11.910930 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 2998b60071cd408aa1cd06fb57d76a8f (table rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]) successfully deleted
I20240501 14:03:11.914455 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:12.146236 5330 raft_consensus.cc:979] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: : Instructing follower 7e7bc120bd74479ab378f779778fd1e8 to start an election
I20240501 14:03:12.146418 5216 raft_consensus.cc:1067] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Signalling peer 7e7bc120bd74479ab378f779778fd1e8 to start an election
I20240501 14:03:12.147454 29007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df"
dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
from {username='slave'} at 127.16.36.129:37221
I20240501 14:03:12.147716 29007 raft_consensus.cc:482] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.147835 29007 raft_consensus.cc:3037] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:12.151288 29007 raft_consensus.cc:504] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:12.153282 30928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a0e901103604414c829b40edf9f24c28"
I20240501 14:03:12.153555 30928 raft_consensus.cc:3037] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:12.155794 28745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0bf08cbb6d347d783b1fa7762c802df" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:03:12.157007 30928 raft_consensus.cc:2454] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 2.
I20240501 14:03:12.158087 28942 leader_election.cc:302] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, a0e901103604414c829b40edf9f24c28; no voters:
I20240501 14:03:12.158587 5083 raft_consensus.cc:2781] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:12.160706 29007 leader_election.cc:288] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 2 election: Requested vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:03:12.162674 28745 raft_consensus.cc:3032] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:12.162813 28745 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:03:12.163199 28745 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:12.163681 28745 raft_consensus.cc:3037] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:12.168318 28745 raft_consensus.cc:2454] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 2.
I20240501 14:03:12.207087 5083 raft_consensus.cc:686] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 2 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:03:12.207544 5083 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:12.208070 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:12.222499 28598 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 1 to 2, leader changed from 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 2 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:12.229192 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:12.270033 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:12.274536 29138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312"
dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59970
I20240501 14:03:12.274825 29138 raft_consensus.cc:595] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:12.278349 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:12.290402 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:12.296350 29118 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:12.358305 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:12.366690 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:12.382231 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:12.388783 29118 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:12.412204 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:12.420154 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:12.435959 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:12.456691 30908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:12.470314 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:12.512950 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:12.569473 5204 raft_consensus.cc:979] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: : Instructing follower a0e901103604414c829b40edf9f24c28 to start an election
I20240501 14:03:12.569667 5204 raft_consensus.cc:1067] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Signalling peer a0e901103604414c829b40edf9f24c28 to start an election
I20240501 14:03:12.570533 30928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312"
dest_uuid: "a0e901103604414c829b40edf9f24c28"
from {username='slave'} at 127.16.36.132:45273
I20240501 14:03:12.570887 30928 raft_consensus.cc:482] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:12.571013 30928 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:12.573871 30928 raft_consensus.cc:504] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:03:12.578644 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:03:12.578879 29138 raft_consensus.cc:3032] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:12.578948 29138 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:03:12.579188 29138 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:03:12.579506 29138 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:12.582849 29138 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
I20240501 14:03:12.583722 30861 leader_election.cc:302] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters:
I20240501 14:03:12.594231 28745 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:12.595113 5083 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:12.596024 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:03:12.596212 29007 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:12.600682 29007 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
I20240501 14:03:12.602278 30928 leader_election.cc:288] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 2 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:03:12.637285 5397 raft_consensus.cc:2781] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 14:03:12.638229 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 52091.8us
W20240501 14:03:12.642992 30861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 59345.5us
I20240501 14:03:12.651716 30928 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:12.652280 5083 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:12.655941 5397 raft_consensus.cc:686] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 2 LEADER]: Becoming Leader. State: Replica: a0e901103604414c829b40edf9f24c28, State: Running, Role: LEADER
I20240501 14:03:12.656884 5397 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:03:12.715453 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:12.757916 29118 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:12.759503 28598 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 reported cstate change: term changed from 1 to 2, leader changed from 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) to a0e901103604414c829b40edf9f24c28 (127.16.36.135). New cstate: current_term: 2 leader_uuid: "a0e901103604414c829b40edf9f24c28" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:03:12.788827 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:12.829990 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:12.889672 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:12.891916 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:12.912495 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:12.935773 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:13.107640 28743 consensus_queue.cc:237] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:13.111315 31059 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.112007 5340 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.112623 30661 raft_consensus.cc:1261] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.113914 5340 consensus_queue.cc:1035] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.118774 5340 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:13.119773 31059 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:13.120851 30661 raft_consensus.cc:2932] T 014635e01a22420b8bc21e879449749b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:13.125870 28598 catalog_manager.cc:5484] T 014635e01a22420b8bc21e879449749b P 826ccf8358b6453d88bcb132290b82a5 reported cstate change: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:13.136334 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 014635e01a22420b8bc21e879449749b with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:13.164444 5497 tablet_replica.cc:330] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:13.165338 5497 raft_consensus.cc:2229] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:13.167376 5497 raft_consensus.cc:2258] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:13.169813 29007 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer a0e901103604414c829b40edf9f24c28: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:13.170857 5497 ts_tablet_manager.cc:1875] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:13.170873 5397 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:13.174140 29138 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer a0e901103604414c829b40edf9f24c28: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:13.174907 5397 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:13.191962 28875 consensus_queue.cc:237] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:13.227473 5497 ts_tablet_manager.cc:1888] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:13.230376 5497 log.cc:1196] T 014635e01a22420b8bc21e879449749b P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/014635e01a22420b8bc21e879449749b
I20240501 14:03:13.231583 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 014635e01a22420b8bc21e879449749b (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
I20240501 14:03:13.279068 30794 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.280175 5233 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.288179 29118 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:13.290092 30661 raft_consensus.cc:1261] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:13.290910 5233 consensus_queue.cc:1035] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:13.323530 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:13.349505 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:13.353495 5525 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:13.359875 30662 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:13.363121 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:13.365753 28601 catalog_manager.cc:5484] T bb712628c7764a138f5425d14b724d9e P 26acba62745c48e3ba8180748d55b211 reported cstate change: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:13.372092 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet bb712628c7764a138f5425d14b724d9e with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59004
W20240501 14:03:13.439081 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 74857.6us
W20240501 14:03:13.441861 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 105066us
I20240501 14:03:13.450026 30794 raft_consensus.cc:2932] T bb712628c7764a138f5425d14b724d9e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:13.455338 5532 tablet_replica.cc:330] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:13.461261 5532 raft_consensus.cc:2229] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:13.461753 5532 raft_consensus.cc:2258] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:13.471139 5532 ts_tablet_manager.cc:1875] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:13.472651 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:13.506305 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:13.533367 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:13.550470 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:13.595207 5532 ts_tablet_manager.cc:1888] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:13.595435 5532 log.cc:1196] T bb712628c7764a138f5425d14b724d9e P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb712628c7764a138f5425d14b724d9e
I20240501 14:03:13.596335 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet bb712628c7764a138f5425d14b724d9e (table rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]) successfully deleted
W20240501 14:03:13.883554 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 103625us
I20240501 14:03:13.889642 29007 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:13.916272 30928 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:13.917137 5347 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:13.922057 5083 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:13.927778 30926 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:13.949510 28601 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 7 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 2 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:14.007308 28725 tablet_service.cc:1514] Processing DeleteTablet for tablet e0bf08cbb6d347d783b1fa7762c802df with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46536
I20240501 14:03:14.007233 29007 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
W20240501 14:03:14.011397 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0bf08cbb6d347d783b1fa7762c802df. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:14.011368 30928 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:14.013833 5347 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:14.018740 5347 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:14.020172 5634 tablet_replica.cc:330] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:03:14.021046 5634 raft_consensus.cc:2229] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:14.023380 30928 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:14.024725 5634 raft_consensus.cc:2258] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:14.027143 5634 ts_tablet_manager.cc:1875] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:14.027382 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e0bf08cbb6d347d783b1fa7762c802df with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:14.030478 28601 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 9 to 10, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 2 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:14.062173 5634 ts_tablet_manager.cc:1888] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:03:14.062732 5634 log.cc:1196] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/e0bf08cbb6d347d783b1fa7762c802df
I20240501 14:03:14.063906 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet e0bf08cbb6d347d783b1fa7762c802df (table rebalance_test_table_2 [id=7974accec32844bfa75c66b42b48e0eb]) successfully deleted
I20240501 14:03:14.348589 30928 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:03:14.353065 29007 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer a0e901103604414c829b40edf9f24c28: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:14.353739 5588 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:14.421358 5734 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:03:14.423789 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:14.424877 29007 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:03:14.474387 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.484004 28598 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 7 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 2 leader_uuid: "a0e901103604414c829b40edf9f24c28" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:03:14.492348 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b3db79ff754e8bbe9e5865f7104312 with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:14.525018 5759 tablet_replica.cc:330] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:14.525357 5759 raft_consensus.cc:2229] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:14.525689 5759 raft_consensus.cc:2258] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:14.538966 5759 ts_tablet_manager.cc:1875] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:14.542347 29118 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:14.545214 30642 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:03:14.565977 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:14.577371 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:14.578827 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:14.583213 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:14.598943 29118 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:14.616696 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:14.625501 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:14.656186 5759 ts_tablet_manager.cc:1888] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:03:14.665822 5759 log.cc:1196] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/60b3db79ff754e8bbe9e5865f7104312
I20240501 14:03:14.667004 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 60b3db79ff754e8bbe9e5865f7104312 (table rebalance_test_table_0 [id=c026765fe9274ecab686de9c27c27e59]) successfully deleted
I20240501 14:03:14.681702 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:14.700332 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:14.702147 30928 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:14.717496 29007 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer a0e901103604414c829b40edf9f24c28: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:14.723389 5734 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:14.726336 5588 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:14.732127 29007 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:14.735430 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:14.738667 28598 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 9 to 10, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 2 leader_uuid: "a0e901103604414c829b40edf9f24c28" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:14.749029 5770 ts_tablet_manager.cc:902] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:03:14.770421 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 60b3db79ff754e8bbe9e5865f7104312 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:14.775624 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:14.777956 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60b3db79ff754e8bbe9e5865f7104312. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:14.798512 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:14.909864 5770 tablet_copy_client.cc:308] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:03:14.910760 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet e0bf08cbb6d347d783b1fa7762c802df from peer 26acba62745c48e3ba8180748d55b211 ({username='slave'} at 127.16.36.133:52377)
I20240501 14:03:14.910965 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet e0bf08cbb6d347d783b1fa7762c802df from peer 26acba62745c48e3ba8180748d55b211 at {username='slave'} at 127.16.36.133:52377: session id = 26acba62745c48e3ba8180748d55b211-e0bf08cbb6d347d783b1fa7762c802df
I20240501 14:03:14.917667 29027 tablet_copy_source_session.cc:205] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:14.920373 5770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bf08cbb6d347d783b1fa7762c802df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:14.936784 5770 tablet_copy_client.cc:783] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:14.937288 5770 tablet_copy_client.cc:647] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:14.941289 5770 tablet_copy_client.cc:523] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:14.945521 5770 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: Bootstrap starting.
W20240501 14:03:15.055841 30729 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 53623us
W20240501 14:03:15.059226 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 43928.8us
I20240501 14:03:15.064977 5770 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:15.065723 5770 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: Bootstrap complete.
I20240501 14:03:15.066035 5770 ts_tablet_manager.cc:1367] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: Time spent bootstrapping tablet: real 0.121s user 0.056s sys 0.007s
I20240501 14:03:15.067073 5770 raft_consensus.cc:348] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:15.067314 5770 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Initialized, Role: FOLLOWER
I20240501 14:03:15.067564 5770 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:15.076002 5770 ts_tablet_manager.cc:1398] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240501 14:03:15.079406 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 26acba62745c48e3ba8180748d55b211-e0bf08cbb6d347d783b1fa7762c802df received from {username='slave'} at 127.16.36.133:52377
I20240501 14:03:15.079543 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 26acba62745c48e3ba8180748d55b211-e0bf08cbb6d347d783b1fa7762c802df on tablet e0bf08cbb6d347d783b1fa7762c802df with peer 26acba62745c48e3ba8180748d55b211
I20240501 14:03:15.108264 29138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5"
dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60082
I20240501 14:03:15.108507 29138 raft_consensus.cc:595] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:15.134795 5821 ts_tablet_manager.cc:902] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:03:15.144188 28940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 48499us
I20240501 14:03:15.222396 5821 tablet_copy_client.cc:308] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.135:42127
I20240501 14:03:15.230437 5204 raft_consensus.cc:979] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: : Instructing follower 26acba62745c48e3ba8180748d55b211 to start an election
I20240501 14:03:15.232671 5204 raft_consensus.cc:1067] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Signalling peer 26acba62745c48e3ba8180748d55b211 to start an election
I20240501 14:03:15.233897 30662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5"
dest_uuid: "26acba62745c48e3ba8180748d55b211"
from {username='slave'} at 127.16.36.132:41411
I20240501 14:03:15.234313 30662 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:15.234524 30662 raft_consensus.cc:3037] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:15.238914 30662 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:15.239815 30948 tablet_copy_service.cc:140] P a0e901103604414c829b40edf9f24c28: Received BeginTabletCopySession request for tablet 60b3db79ff754e8bbe9e5865f7104312 from peer 826ccf8358b6453d88bcb132290b82a5 ({username='slave'} at 127.16.36.136:44609)
I20240501 14:03:15.240092 30948 tablet_copy_service.cc:161] P a0e901103604414c829b40edf9f24c28: Beginning new tablet copy session on tablet 60b3db79ff754e8bbe9e5865f7104312 from peer 826ccf8358b6453d88bcb132290b82a5 at {username='slave'} at 127.16.36.136:44609: session id = 826ccf8358b6453d88bcb132290b82a5-60b3db79ff754e8bbe9e5865f7104312
I20240501 14:03:15.241609 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5" candidate_uuid: "26acba62745c48e3ba8180748d55b211" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:03:15.241916 29138 raft_consensus.cc:3032] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:15.241999 29138 raft_consensus.cc:729] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:03:15.242322 29138 consensus_queue.cc:260] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:15.242897 29138 raft_consensus.cc:3037] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:15.245831 29138 raft_consensus.cc:2454] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26acba62745c48e3ba8180748d55b211 in term 2.
I20240501 14:03:15.246069 30948 tablet_copy_source_session.cc:205] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:15.246521 30595 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26acba62745c48e3ba8180748d55b211, 3a3972b170c54344a4ba61f176f837ba; no voters:
I20240501 14:03:15.247040 5817 raft_consensus.cc:2781] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:15.248848 5821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b3db79ff754e8bbe9e5865f7104312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:15.248941 30661 raft_consensus.cc:1203] T e0bf08cbb6d347d783b1fa7762c802df P 26acba62745c48e3ba8180748d55b211 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 14:03:15.259647 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7c7b963588436bb778e45da469c7d5" candidate_uuid: "26acba62745c48e3ba8180748d55b211" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
I20240501 14:03:15.259922 28875 raft_consensus.cc:3037] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:15.263760 28875 raft_consensus.cc:2454] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26acba62745c48e3ba8180748d55b211 in term 2.
I20240501 14:03:15.264722 30662 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [CANDIDATE]: Term 2 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:03:15.315366 5817 raft_consensus.cc:686] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 2 LEADER]: Becoming Leader. State: Replica: 26acba62745c48e3ba8180748d55b211, State: Running, Role: LEADER
I20240501 14:03:15.316172 5817 consensus_queue.cc:237] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:15.372363 5821 tablet_copy_client.cc:783] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:15.372761 5821 tablet_copy_client.cc:647] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:15.375360 5821 tablet_copy_client.cc:523] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:15.435536 5821 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap starting.
I20240501 14:03:15.453675 28598 catalog_manager.cc:5484] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 reported cstate change: term changed from 1 to 2, leader changed from 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) to 26acba62745c48e3ba8180748d55b211 (127.16.36.133). New cstate: current_term: 2 leader_uuid: "26acba62745c48e3ba8180748d55b211" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:15.469655 29138 consensus_queue.cc:237] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:15.471082 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:15.480159 30661 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:15.481175 5204 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:15.489848 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:15.491912 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:15.497058 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 45777.7us
W20240501 14:03:15.497357 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 48674.9us
I20240501 14:03:15.500242 30794 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:15.501178 5808 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:15.503291 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:15.551292 31059 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:03:15.552671 5808 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:15.587235 5808 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:15.623262 31059 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 826ccf8358b6453d88bcb132290b82a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:15.623699 28604 catalog_manager.cc:5484] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 7 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:15.626924 30794 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:15.631129 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:15.631870 30661 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:15.639184 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet f4402de0b48041c383568feb9365147e with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:15.641008 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:15.644958 30728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 70585.6us
I20240501 14:03:15.650825 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:15.664991 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:15.666597 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:15.675081 5925 tablet_replica.cc:330] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:15.677752 5925 raft_consensus.cc:2229] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:15.679015 5925 raft_consensus.cc:2258] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:15.681021 5925 ts_tablet_manager.cc:1875] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:15.697365 5925 ts_tablet_manager.cc:1888] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:03:15.697639 5925 log.cc:1196] T f4402de0b48041c383568feb9365147e P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4402de0b48041c383568feb9365147e
I20240501 14:03:15.699529 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet f4402de0b48041c383568feb9365147e (table rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]) successfully deleted
I20240501 14:03:15.769963 29138 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 26acba62745c48e3ba8180748d55b211: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:03:15.770942 5817 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.786341 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:15.829797 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:15.834223 28875 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Refusing update from remote peer 26acba62745c48e3ba8180748d55b211: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:03:15.835022 5865 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:15.857376 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:15.876446 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:15.880340 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:15.931592 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:15.946553 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:16.145402 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:16.166980 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:16.185798 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:16.209996 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:16.247272 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:16.249148 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:16.256459 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:16.281697 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:16.394721 5821 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:16.400238 5821 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: Bootstrap complete.
I20240501 14:03:16.401554 5821 ts_tablet_manager.cc:1367] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: Time spent bootstrapping tablet: real 0.966s user 0.063s sys 0.011s
I20240501 14:03:16.427642 5821 raft_consensus.cc:348] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:16.428256 5821 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826ccf8358b6453d88bcb132290b82a5, State: Initialized, Role: FOLLOWER
I20240501 14:03:16.429955 5821 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:16.433941 5821 ts_tablet_manager.cc:1398] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240501 14:03:16.442883 30948 tablet_copy_service.cc:342] P a0e901103604414c829b40edf9f24c28: Request end of tablet copy session 826ccf8358b6453d88bcb132290b82a5-60b3db79ff754e8bbe9e5865f7104312 received from {username='slave'} at 127.16.36.136:44609
I20240501 14:03:16.443050 30948 tablet_copy_service.cc:433] P a0e901103604414c829b40edf9f24c28: ending tablet copy session 826ccf8358b6453d88bcb132290b82a5-60b3db79ff754e8bbe9e5865f7104312 on tablet 60b3db79ff754e8bbe9e5865f7104312 with peer 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:16.447971 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:16.457828 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:16.506039 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:16.506183 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:16.526232 31059 raft_consensus.cc:1203] T 60b3db79ff754e8bbe9e5865f7104312 P 826ccf8358b6453d88bcb132290b82a5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 14:03:16.528410 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:16.538169 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:16.538226 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:16.547402 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:16.710258 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 46602.8us
I20240501 14:03:16.806890 29006 consensus_queue.cc:237] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:16.820190 30928 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.820979 30661 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:16.822705 5826 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:16.823788 5822 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:03:16.885370 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 61020.1us
I20240501 14:03:16.920148 5822 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:16.933034 28598 catalog_manager.cc:5484] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:16.933873 30662 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
W20240501 14:03:16.951047 30728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 119659us
I20240501 14:03:16.961887 30928 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:16.988242 29006 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:16.989601 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 12510f71b6a742ab8cc359243de8119b with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:17.003368 6097 tablet_replica.cc:330] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:17.003841 6097 raft_consensus.cc:2229] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:17.004213 6097 raft_consensus.cc:2258] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:17.005297 6097 ts_tablet_manager.cc:1875] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:17.007833 30794 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.008435 6073 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.014130 6097 ts_tablet_manager.cc:1888] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:17.014367 6097 log.cc:1196] T 12510f71b6a742ab8cc359243de8119b P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/12510f71b6a742ab8cc359243de8119b
I20240501 14:03:17.014185 5822 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:17.015357 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 12510f71b6a742ab8cc359243de8119b (table rebalance_test_table_4 [id=16d485c2dc3c415b9eb32f71cc8c2632]) successfully deleted
I20240501 14:03:17.022082 30794 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:17.032282 28604 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 6 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:17.034611 30928 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:17.035275 5822 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.037039 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1b3fec785f43c4a3bacedecd8472dd with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:17.095094 6110 tablet_replica.cc:330] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:17.095896 6110 raft_consensus.cc:2229] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:17.095968 30926 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:17.096675 6110 raft_consensus.cc:2258] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:17.097549 6110 ts_tablet_manager.cc:1875] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:17.135134 31059 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:17.139890 6110 ts_tablet_manager.cc:1888] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:03:17.140184 30926 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 826ccf8358b6453d88bcb132290b82a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:17.141172 4420 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:17.141322 6110 log.cc:1196] T ec1b3fec785f43c4a3bacedecd8472dd P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/ec1b3fec785f43c4a3bacedecd8472dd
I20240501 14:03:17.142369 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet ec1b3fec785f43c4a3bacedecd8472dd (table rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]) successfully deleted
I20240501 14:03:17.170564 6137 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:17.178431 30926 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:17.200397 28601 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 reported cstate change: config changed from index 8 to 9, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 2 leader_uuid: "826ccf8358b6453d88bcb132290b82a5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:17.207930 31059 consensus_queue.cc:237] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:17.212270 30662 consensus_queue.cc:237] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:17.214421 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b9db597e654ed48b679abf58b8a2c8 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:17.215132 6097 tablet_replica.cc:330] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:17.215420 6097 raft_consensus.cc:2229] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:17.215653 6097 raft_consensus.cc:2258] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:17.216579 6097 ts_tablet_manager.cc:1875] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:03:17.227807 30995 consensus_peers.cc:485] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8b9db597e654ed48b679abf58b8a2c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:17.228403 28875 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Refusing update from remote peer 26acba62745c48e3ba8180748d55b211: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:17.229104 5817 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:17.241027 6097 ts_tablet_manager.cc:1888] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:03:17.241286 6097 log.cc:1196] T e8b9db597e654ed48b679abf58b8a2c8 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8b9db597e654ed48b679abf58b8a2c8
I20240501 14:03:17.242154 30926 raft_consensus.cc:1261] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 826ccf8358b6453d88bcb132290b82a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:17.242269 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet e8b9db597e654ed48b679abf58b8a2c8 (table rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]) successfully deleted
I20240501 14:03:17.243149 6152 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:17.243985 6137 consensus_queue.cc:1035] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:17.246155 28875 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:17.253645 28598 catalog_manager.cc:5484] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 7, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 2 leader_uuid: "26acba62745c48e3ba8180748d55b211" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:17.261013 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7c7b963588436bb778e45da469c7d5 with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:17.262611 6110 tablet_replica.cc:330] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:17.262884 6110 raft_consensus.cc:2229] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:17.263064 6110 raft_consensus.cc:2258] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:17.263952 6110 ts_tablet_manager.cc:1875] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:17.264766 6137 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:17.275242 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e8b9db597e654ed48b679abf58b8a2c8 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:17.285557 30926 raft_consensus.cc:2932] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:17.293844 28598 catalog_manager.cc:5484] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 2 leader_uuid: "826ccf8358b6453d88bcb132290b82a5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:17.294322 6110 ts_tablet_manager.cc:1888] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:03:17.294474 6110 log.cc:1196] T 9b7c7b963588436bb778e45da469c7d5 P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/9b7c7b963588436bb778e45da469c7d5
I20240501 14:03:17.304875 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 9b7c7b963588436bb778e45da469c7d5 (table rebalance_test_table_8 [id=4ce17c35a7b64e0097eaee91ab51749f]) successfully deleted
I20240501 14:03:17.334244 30662 consensus_queue.cc:237] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:17.338457 28875 raft_consensus.cc:1261] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Refusing update from remote peer 26acba62745c48e3ba8180748d55b211: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:17.339313 5817 consensus_queue.cc:1035] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:17.342496 5817 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:17.344785 28875 raft_consensus.cc:2932] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:17.356753 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9b7c7b963588436bb778e45da469c7d5 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:17.359470 28598 catalog_manager.cc:5484] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 reported cstate change: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) added. New cstate: current_term: 2 leader_uuid: "26acba62745c48e3ba8180748d55b211" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
W20240501 14:03:17.373728 30596 consensus_peers.cc:485] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b7c7b963588436bb778e45da469c7d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:17.708238 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:17.747632 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:17.769177 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:17.773568 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:17.798677 31041 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:17.879554 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:17.923095 6258 ts_tablet_manager.cc:902] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
I20240501 14:03:17.924260 6259 ts_tablet_manager.cc:902] T 9b7c7b963588436bb778e45da469c7d5 P 826ccf8358b6453d88bcb132290b82a5: Initiating tablet copy from peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
F20240501 14:03:17.924708 6259 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-014635e01a22420b8bc21e879449749b.queue_time_us, tablet-download-pool-014635e01a22420b8bc21e879449749b.queue_time_us) (tablet-download-pool-014635e01a22420b8bc21e879449749b.run_cpu_time_us, tablet-download-pool-014635e01a22420b8bc21e879449749b.run_cpu_time_us) (tablet-download-pool-014635e01a22420b8bc21e879449749b.run_wall_time_us, tablet-download-pool-014635e01a22420b8bc21e879449749b.run_wall_time_us) (tablet-download-pool-111ba172d7e44054872c8fc8cac8ba88.queue_time_us, tablet-download-pool-111ba172d7e44054872c8fc8cac8ba88.queue_time_us) (tablet-download-pool-111ba172d7e44054872c8fc8cac8ba88.run_cpu_time_us, tablet-download-pool-111ba172d7e44054872c8fc8cac8ba88.run_cpu_time_us) (tablet-download-pool-111ba172d7e44054872c8fc8cac8ba88.run_wall_time_us, tablet-download-pool-111ba172d7e44054872c8fc8cac8ba88.run_wall_time_us) (tablet-download-pool-135da52054584a27959efdd9126cfcef.queue_time_us, tablet-download-pool-135da52054584a27959efdd9126cfcef.queue_time_us) (tablet-download-pool-135da52054584a27959efdd9126cfcef.run_cpu_time_us, tablet-download-pool-135da52054584a27959efdd9126cfcef.run_cpu_time_us) (tablet-download-pool-135da52054584a27959efdd9126cfcef.run_wall_time_us, tablet-download-pool-135da52054584a27959efdd9126cfcef.run_wall_time_us) (tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.queue_time_us, tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.queue_time_us) (tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_cpu_time_us, tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_cpu_time_us) (tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_wall_time_us, tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_wall_time_us) (tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.queue_time_us, tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.queue_time_us) (tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_cpu_time_us, tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_cpu_time_us) (tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_wall_time_us, tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_wall_time_us) (tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.queue_time_us, tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.queue_time_us) (tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_cpu_time_us, tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_cpu_time_us) (tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_wall_time_us, tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_wall_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.queue_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.queue_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_cpu_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_cpu_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_wall_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_wall_time_us) (tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.queue_time_us, tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.queue_time_us) (tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.run_cpu_time_us, tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.run_cpu_time_us) (tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.run_wall_time_us, tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.run_wall_time_us) (tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.queue_time_us, tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.queue_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.queue_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.queue_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_cpu_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_cpu_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_wall_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_wall_time_us) (tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.queue_time_us, tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.queue_time_us) (tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_cpu_time_us, tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_cpu_time_us) (tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_wall_time_us, tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_wall_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.queue_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.queue_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_cpu_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_cpu_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_wall_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_wall_time_us) (tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.queue_time_us, tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.queue_time_us) (tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_cpu_time_us, tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_cpu_time_us) (tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_wall_time_us, tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_wall_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.queue_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.queue_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.run_cpu_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.run_cpu_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.run_wall_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.run_wall_time_us) (tablet-download-pool-fb04ec77da194832aa63d69aba307de3.queue_time_us, tablet-download-pool-fb04ec77da194832aa63d69aba307de3.queue_time_us) (tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_cpu_time_us, tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_cpu_time_us) (tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_wall_time_us, tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 14:03:17.973332 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7fbb48f34dcd google::LogMessage::Fail() at ??:0
@ 0x7fbb48f38b93 google::LogMessage::SendToLog() at ??:0
I20240501 14:03:18.011761 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7fbb48f347cc google::LogMessage::Flush() at ??:0
@ 0x7fbb48f35f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:03:18.043188 6258 tablet_copy_client.cc:308] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.136:36399
I20240501 14:03:18.044337 31081 tablet_copy_service.cc:140] P 826ccf8358b6453d88bcb132290b82a5: Received BeginTabletCopySession request for tablet e8b9db597e654ed48b679abf58b8a2c8 from peer 8fc5d8222c164a679e7a99966da1eaaf ({username='slave'} at 127.16.36.134:56971)
I20240501 14:03:18.044628 31081 tablet_copy_service.cc:161] P 826ccf8358b6453d88bcb132290b82a5: Beginning new tablet copy session on tablet e8b9db597e654ed48b679abf58b8a2c8 from peer 8fc5d8222c164a679e7a99966da1eaaf at {username='slave'} at 127.16.36.134:56971: session id = 8fc5d8222c164a679e7a99966da1eaaf-e8b9db597e654ed48b679abf58b8a2c8
@ 0x7fbb4bc75742 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:03:18.049937 31081 tablet_copy_source_session.cc:205] T e8b9db597e654ed48b679abf58b8a2c8 P 826ccf8358b6453d88bcb132290b82a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:18.052708 6258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b9db597e654ed48b679abf58b8a2c8. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fbb4bc41f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fbb4bc3bcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fbb5b08e8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fbb5b0a6daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fbb5b1d1480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fbb5b209a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fbb5b1ec981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fbb5b1ec70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fbb4bc45b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbb4bc4919c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbb4bc49019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fbb4bc2401e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbb4efa96db start_thread at ??:0
@ 0x7fbb47de771f clone at ??:0
I20240501 14:03:18.165539 6258 tablet_copy_client.cc:783] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:18.165774 6258 tablet_copy_client.cc:647] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:18.224609 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:18.242102 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:18.330024 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:18.436980 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:03:18.479753 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 54333.2us
I20240501 14:03:18.524940 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.537261 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:18.558240 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:18.658691 28986 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:03:18.666893 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:18.667762 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:18.699699 30775 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:03:18.715533 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:18.759353 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:18.764842 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:18.877868 30862 connection.cc:535] client connection to 127.16.36.136:36399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.878136 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.896343 30729 connection.cc:535] server connection from 127.16.36.136:43619 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.896858 30596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.36.136:36399 (error 108)
W20240501 14:03:18.907555 28679 connection.cc:535] client connection to 127.16.36.136:36399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.907820 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.918872 28941 connection.cc:535] client connection to 127.16.36.136:36399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.919166 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.924093 30729 connection.cc:535] client connection to 127.16.36.136:36399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.924160 28810 connection.cc:535] client connection to 127.16.36.136:36399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.924378 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.924377 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.924949 29072 connection.cc:535] client connection to 127.16.36.136:36399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.925226 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:18.935151 6396 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:03:18.891611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.136:36399 (local address 127.16.36.135:42559)
0501 14:03:18.900573 (+ 8962us) negotiation.cc:107] Waiting for socket to connect
0501 14:03:18.900603 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 14:03:18.900773 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:03:18.934268 (+ 33495us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":8793,"thread_start_us":9246,"threads_started":1}
W20240501 14:03:18.935438 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.946218 6401 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:03:18.911815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.136:36399 (local address 127.16.36.133:40945)
0501 14:03:18.945076 (+ 33261us) negotiation.cc:107] Waiting for socket to connect
0501 14:03:18.945248 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33111,"threads_started":1}
W20240501 14:03:18.946662 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection reset by peer (error 104)
W20240501 14:03:18.963593 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.963760 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.963853 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.963768 6258 ts_tablet_manager.cc:980] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: Tablet Copy: Unable to fetch data from remote peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Unable to download WAL segment with seq. number 1: unable to fetch data from remote: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:03:18.963979 6258 tablet_replica.cc:330] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: stopping tablet replica
I20240501 14:03:18.964120 6258 raft_consensus.cc:2229] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:18.964236 6258 raft_consensus.cc:2258] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 14:03:18.968870 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.969046 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.969110 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.969379 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.972579 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.972738 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:18.974705 6258 tablet_copy_client.cc:1108] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:18.975448 6258 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Already present: tried to load directory group for tablet e8b9db597e654ed48b679abf58b8a2c8 but one is already registered
I20240501 14:03:18.975695 6258 ts_tablet_manager.cc:1875] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:03:18.995890 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.001170 6258 ts_tablet_manager.cc:1888] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20240501 14:03:19.001384 6258 log.cc:1196] T e8b9db597e654ed48b679abf58b8a2c8 P 8fc5d8222c164a679e7a99966da1eaaf: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-5/wal/wals/e8b9db597e654ed48b679abf58b8a2c8
I20240501 14:03:19.045459 28745 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:19.059499 28874 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:19.059784 29006 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd"
dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57358
I20240501 14:03:19.059989 29006 raft_consensus.cc:595] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:19.060266 6226 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:03:19.063057 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.079128 6374 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:19.083724 28875 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:19.091226 28604 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } }
I20240501 14:03:19.181465 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet 135da52054584a27959efdd9126cfcef with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:19.214066 6471 tablet_replica.cc:330] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:19.218838 6471 raft_consensus.cc:2229] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:19.221240 6471 raft_consensus.cc:2258] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:19.228564 6471 ts_tablet_manager.cc:1875] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:03:19.234005 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:19.243139 6471 ts_tablet_manager.cc:1888] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:03:19.243294 6471 log.cc:1196] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/135da52054584a27959efdd9126cfcef
I20240501 14:03:19.244387 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 135da52054584a27959efdd9126cfcef (table rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]) successfully deleted
I20240501 14:03:19.246428 6444 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:19.246639 6444 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:19.247695 6444 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:19.267249 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:19.268365 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:03:19.269143 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:19.269281 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:19.270159 6444 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:19.418684 29138 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:19.431993 30794 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:19.433009 6040 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:19.448362 6040 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:19.469732 6542 raft_consensus.cc:979] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: : Instructing follower 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:19.471810 6542 raft_consensus.cc:1067] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Signalling peer 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:19.472751 30794 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd"
dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf"
from {username='slave'} at 127.16.36.131:45889
I20240501 14:03:19.473134 30794 raft_consensus.cc:482] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:19.473331 30794 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:19.481557 28604 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:19.485991 30794 raft_consensus.cc:504] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:19.489511 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:03:19.489842 29006 raft_consensus.cc:3032] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:19.489930 29006 raft_consensus.cc:729] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:03:19.490260 29006 consensus_queue.cc:260] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:19.491230 29006 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:19.497403 28856 tablet_service.cc:1514] Processing DeleteTablet for tablet 2020fe37059a4e74bb45190398e4e177 with delete_type TABLET_DATA_TOMBSTONED (TS cd3c04ebb6c84b15a1b3adc748737094 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37770
I20240501 14:03:19.499408 29006 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:19.500021 29138 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:19.500774 30729 leader_election.cc:302] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, 8fc5d8222c164a679e7a99966da1eaaf; no voters:
I20240501 14:03:19.501245 6074 raft_consensus.cc:2781] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Leader election won for term 2
W20240501 14:03:19.511298 29073 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2020fe37059a4e74bb45190398e4e177. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:19.512909 6074 raft_consensus.cc:686] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Becoming Leader. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Running, Role: LEADER
I20240501 14:03:19.513478 6074 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:19.516500 30793 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:03:19.519033 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a0e901103604414c829b40edf9f24c28"
I20240501 14:03:19.519294 30926 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:19.524554 6551 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:19.526523 30926 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:19.526404 30793 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:19.534644 6549 tablet_replica.cc:330] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:03:19.534896 6549 raft_consensus.cc:2229] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:19.535094 6549 raft_consensus.cc:2258] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:19.535840 6549 ts_tablet_manager.cc:1875] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:19.537993 30794 leader_election.cc:288] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Requested vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:03:19.538337 6040 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:19.539137 28604 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: term changed from 1 to 2, leader changed from 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) to 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134). New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:19.549592 6549 ts_tablet_manager.cc:1888] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:19.549815 6549 log.cc:1196] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/wals/2020fe37059a4e74bb45190398e4e177
I20240501 14:03:19.550701 28588 catalog_manager.cc:4830] TS cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): tablet 2020fe37059a4e74bb45190398e4e177 (table rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]) successfully deleted
I20240501 14:03:19.558568 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2020fe37059a4e74bb45190398e4e177 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:19.561239 30793 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:19.570191 28604 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 6 to 7, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:19.786794 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:19.824052 30596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 51560.2us
I20240501 14:03:19.828850 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:19.829769 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:19.837445 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 65228.1us
I20240501 14:03:19.865226 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:19.883903 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:19.903044 30927 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:19.903739 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:19.904104 6074 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:19.934578 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:19.944515 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:19.945556 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:19.970867 6661 ts_tablet_manager.cc:902] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:03:20.013278 6661 tablet_copy_client.cc:308] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:03:20.019932 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:20.021154 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:20.022272 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 2020fe37059a4e74bb45190398e4e177 from peer a0e901103604414c829b40edf9f24c28 ({username='slave'} at 127.16.36.135:53567)
I20240501 14:03:20.022445 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 2020fe37059a4e74bb45190398e4e177 from peer a0e901103604414c829b40edf9f24c28 at {username='slave'} at 127.16.36.135:53567: session id = a0e901103604414c829b40edf9f24c28-2020fe37059a4e74bb45190398e4e177
I20240501 14:03:20.025524 29158 tablet_copy_source_session.cc:205] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:20.026021 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:20.038147 6661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2020fe37059a4e74bb45190398e4e177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:20.044169 29006 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:20.050114 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:20.050360 6074 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:20.082278 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:20.082558 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:20.173316 6661 tablet_copy_client.cc:783] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:20.174913 6661 tablet_copy_client.cc:647] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:20.176990 6661 tablet_copy_client.cc:523] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:20.205207 6661 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: Bootstrap starting.
I20240501 14:03:20.273599 6661 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:20.274435 6661 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: Bootstrap complete.
I20240501 14:03:20.274809 6661 ts_tablet_manager.cc:1367] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: Time spent bootstrapping tablet: real 0.077s user 0.058s sys 0.006s
I20240501 14:03:20.275559 6661 raft_consensus.cc:348] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:20.275790 6661 raft_consensus.cc:729] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e901103604414c829b40edf9f24c28, State: Initialized, Role: FOLLOWER
I20240501 14:03:20.276048 6661 consensus_queue.cc:260] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:20.276721 6661 ts_tablet_manager.cc:1398] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:20.281872 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session a0e901103604414c829b40edf9f24c28-2020fe37059a4e74bb45190398e4e177 received from {username='slave'} at 127.16.36.135:53567
I20240501 14:03:20.282006 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session a0e901103604414c829b40edf9f24c28-2020fe37059a4e74bb45190398e4e177 on tablet 2020fe37059a4e74bb45190398e4e177 with peer a0e901103604414c829b40edf9f24c28
W20240501 14:03:20.405088 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:20.522294 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:20.553133 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:20.561435 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:20.562599 30926 raft_consensus.cc:1203] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:03:20.640009 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:20.668815 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:20.743695 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:20.749305 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:20.792268 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:21.060240 29006 consensus_queue.cc:237] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:03:21.072827 30662 raft_consensus.cc:1261] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:03:21.074026 6300 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:21.079304 28745 raft_consensus.cc:1261] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:03:21.083119 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:21.086010 6300 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:03:21.089972 6300 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:21.099737 30662 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:21.106313 28597 catalog_manager.cc:5484] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 26acba62745c48e3ba8180748d55b211 reported cstate change: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
W20240501 14:03:21.118467 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:21.154922 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:21.158423 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet e9bb3eaf3f0e40f8902b9426a845d9b2 with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:21.177853 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:21.172180 6863 tablet_replica.cc:330] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:21.198926 6863 raft_consensus.cc:2229] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:21.200531 6863 raft_consensus.cc:2258] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:21.201442 6863 ts_tablet_manager.cc:1875] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:03:21.202656 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:21.230749 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:21.237716 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:21.246701 6863 ts_tablet_manager.cc:1888] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:03:21.246909 6863 log.cc:1196] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/e9bb3eaf3f0e40f8902b9426a845d9b2
I20240501 14:03:21.250850 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet e9bb3eaf3f0e40f8902b9426a845d9b2 (table rebalance_test_table_3 [id=42106fd27f0b49b2890a9aaaf268a5e3]) successfully deleted
I20240501 14:03:21.252956 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:21.258085 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:21.265233 28745 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:21.295416 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:21.306298 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:21.313916 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:21.318814 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:21.353494 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:21.400000 6444 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:21.400265 6444 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:21.401173 6444 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:03:21.404039 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:21.405623 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:21.406430 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:03:21.406601 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:21.407217 6444 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:03:21.428516 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:21.465426 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:21.480525 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:21.524484 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:21.525763 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:21.529515 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:03:21.540959 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:21.551781 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:21.604696 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:21.605376 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:21.608515 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:21.610008 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:21.671917 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:22.052000 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:22.335407 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:22.345080 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:22.356974 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:22.362371 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:22.380642 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:22.384622 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 175700us
I20240501 14:03:22.385515 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:22.386437 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:22.410313 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:22.440320 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:22.475302 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:22.484747 30908 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:22.484938 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:22.485462 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:22.551295 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:22.624488 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:22.703011 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:22.712364 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:22.713125 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:22.735314 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:22.785080 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:22.786056 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:22.805111 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:23.091454 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:23.141096 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 72924.3us
I20240501 14:03:23.148752 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:23.157301 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:23.218330 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:23.235363 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:23.274240 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:23.301175 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:23.301350 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:23.339527 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:23.362887 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:23.396735 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:03:23.397646 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 87432.1us
W20240501 14:03:23.409992 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:23.427176 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:23.444882 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:23.453253 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:23.485036 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:23.496243 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:23.500723 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:23.567288 30729 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 63732.9us
W20240501 14:03:23.569000 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:23.654650 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:23.824628 29006 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8"
dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57564
I20240501 14:03:23.824874 29006 raft_consensus.cc:595] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:23.830915 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:23.831432 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:23.831516 30775 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:23.841576 31112 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task1g_oHS/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:02:27.938997 31116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:27.939441 31116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:27.961154 31116 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:32.316880 31116 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:02:34.139569 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:34.140030 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:34.140192 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:34.140394 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:34.140636 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:34.140833 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:34.140987 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:34.141129 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:34.144960 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:34.145174 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:34.145380 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:34.145493 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:34.145654 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:34.145778 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:34.145881 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:34.878273 31116 rebalancer_tool.cc:1074] tablet f516f01e242d4b8789130201f748f3ea: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:35.351859 31116 rebalancer_tool.cc:1074] tablet efe7fed8e5d44e8a80ffbdc27619a74d: '343a85ba3b514663af5320cd14a9e144' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:36.158556 31116 rebalancer_tool.cc:1074] tablet 9b473e10a8ad4bd58bb44aabd9656e08: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:36.808533 31116 rebalancer_tool.cc:1074] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:37.231014 31116 rebalancer_tool.cc:1074] tablet e0bf08cbb6d347d783b1fa7762c802df: '3a3972b170c54344a4ba61f176f837ba' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:37.328545 31116 rebalancer_tool.cc:1084] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move ignored: Illegal state: tablet 6feda9b53e484e7ea40a5890b7d51ac4: replica 826ccf8358b6453d88bcb132290b82a5 is already present
I20240501 14:02:37.328739 31116 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:38.605041 32754 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.16.36.190:41971, user_credentials={real_user=slave}} blocked reactor thread for 50412.7us
W20240501 14:02:39.726330 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.734158 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:39.734345 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:39.735047 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:39.735698 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:39.735913 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.736021 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:39.739373 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.739506 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:39.739617 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:39.739717 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:39.739825 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:39.739984 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:39.740182 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
I20240501 14:02:39.940024 31116 rebalancer_tool.cc:1074] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:02:40.615499 31116 rebalancer_tool.cc:1084] tablet 6aaf52ee71b44511b7b8909711ab0a42: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:40.615621 31116 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:42.476447 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:42.476877 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:42.477077 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:42.477236 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:42.477365 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:42.482263 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:42.482586 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:42.486856 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:42.487041 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:42.487149 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:42.487262 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:42.632612 31116 rebalancer_tool.cc:1074] tablet 6aaf52ee71b44511b7b8909711ab0a42: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:02:43.314342 31116 rebalancer_tool.cc:1084] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240501 14:02:43.314497 31116 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:45.881094 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:45.882397 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 826ccf8358b6453d88bcb132290b82a5 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:45.883270 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:45.883821 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:45.884258 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:45.884645 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:45.884956 31116 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
I20240501 14:02:46.194833 31116 rebalancer_tool.cc:1074] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:02:46.413319 31116 rebalancer_tool.cc:1074] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:02:46.576879 31116 rebalancer_tool.cc:1074] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
W20240501 14:03:07.256374 4409 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.16.36.133:40601, user_credentials={real_user=slave}} blocked reactor thread for 65925.8us
I20240501 14:03:08.733180 31116 rebalancer_tool.cc:1136] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:08.930339 31116 rebalancer_tool.cc:1136] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:09.018060 31116 rebalancer_tool.cc:1074] tablet 60b3db79ff754e8bbe9e5865f7104312: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:09.132455 31116 rebalancer_tool.cc:1136] tablet 60b3db79ff754e8bbe9e5865f7104312: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:09.271222 31116 rebalancer_tool.cc:1136] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:09.396922 31116 rebalancer_tool.cc:1136] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:09.612169 31116 rebalancer_tool.cc:1136] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:09.705118 31116 rebalancer_tool.cc:1136] tablet 6aaf52ee71b44511b7b8909711ab0a42: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:09.980190 31116 rebalancer_tool.cc:1136] tablet f516f01e242d4b8789130201f748f3ea: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:10.120066 31116 rebalancer_tool.cc:1136] tablet efe7fed8e5d44e8a80ffbdc27619a74d: '343a85ba3b514663af5320cd14a9e144' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:10.200564 31116 rebalancer_tool.cc:1136] tablet 9b473e10a8ad4bd58bb44aabd9656e08: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:10.341147 31116 rebalancer_tool.cc:1136] tablet e0bf08cbb6d347d783b1fa7762c802df: '3a3972b170c54344a4ba61f176f837ba' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:10.402071 31116 rebalancer_tool.cc:1074] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:10.592020 31116 rebalancer_tool.cc:1074] tablet 12510f71b6a742ab8cc359243de8119b: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:03:10.782971 31116 rebalancer_tool.cc:1074] tablet 2020fe37059a4e74bb45190398e4e177: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:10.893535 31116 rebalancer_tool.cc:1074] tablet 9b7c7b963588436bb778e45da469c7d5: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:11.002015 31116 rebalancer_tool.cc:1074] tablet bb712628c7764a138f5425d14b724d9e: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
W20240501 14:03:18.936769 6340 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:03:18.528101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.136:36399 (local address 127.0.0.1:42644)
0501 14:03:18.528468 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0501 14:03:18.528488 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0501 14:03:18.530792 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:03:18.935691 (+404899us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":161}
I20240501 14:03:19.582737 31116 rebalancer_tool.cc:1136] tablet 12510f71b6a742ab8cc359243de8119b: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:19.666736 31116 rebalancer_tool.cc:1136] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:19.799840 31116 rebalancer_tool.cc:1074] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:03:20.116923 31116 rebalancer_tool.cc:1074] tablet 111ba172d7e44054872c8fc8cac8ba88: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:20.251121 31116 rebalancer_tool.cc:1136] tablet 111ba172d7e44054872c8fc8cac8ba88: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:20.418367 31116 rebalancer_tool.cc:1136] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:20.510372 31116 rebalancer_tool.cc:1136] tablet bb712628c7764a138f5425d14b724d9e: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:20.689002 31116 rebalancer_tool.cc:1136] tablet 9b7c7b963588436bb778e45da469c7d5: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:20.793874 31116 rebalancer_tool.cc:1136] tablet 2020fe37059a4e74bb45190398e4e177: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:21.034269 31116 rebalancer_tool.cc:1074] tablet de161b10c84f4bcc85f8cf7b199f99db: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:21.296223 31116 rebalancer_tool.cc:1136] tablet de161b10c84f4bcc85f8cf7b199f99db: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
Illegal state: tablet server 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): unacceptable health status UNAVAILABLE
I20240501 14:03:23.844100 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:23.861011 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:03:23.896562 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:23.937995 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:23.938313 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:03:23.945856 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:03:23.945909 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:03:23.949733 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:23.949786 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:03:23.966866 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:23.968209 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:23.968489 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:24.003343 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:03:24.045367 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:03:24.051407 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:24.073899 7182 raft_consensus.cc:979] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: : Instructing follower 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:24.074093 7182 raft_consensus.cc:1067] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Signalling peer 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:24.075125 30793 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8"
dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf"
from {username='slave'} at 127.16.36.131:45889
I20240501 14:03:24.075412 30793 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:24.075537 30793 raft_consensus.cc:3037] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:03:24.081087 30596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:03:24.084208 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:24.085572 30793 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:24.087446 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:03:24.087730 29136 raft_consensus.cc:3037] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:24.094348 29136 raft_consensus.cc:2454] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:24.096467 30728 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 8fc5d8222c164a679e7a99966da1eaaf; no voters:
I20240501 14:03:24.096949 6074 raft_consensus.cc:2781] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:24.097108 6074 raft_consensus.cc:686] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Becoming Leader. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Running, Role: LEADER
I20240501 14:03:24.097470 6074 consensus_queue.cc:237] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:24.110024 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8c435795a04b1b90b6e2ff914676c8" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:03:24.110368 29006 raft_consensus.cc:3032] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:24.110450 29006 raft_consensus.cc:729] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:03:24.112008 29006 consensus_queue.cc:260] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:24.112473 29006 raft_consensus.cc:3037] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:03:24.117076 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:24.119469 29006 raft_consensus.cc:2454] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:24.121131 30793 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:03:24.131314 28604 catalog_manager.cc:5484] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: term changed from 1 to 2, leader changed from 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) to 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134). New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
W20240501 14:03:24.194823 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:24.292521 28874 consensus_queue.cc:237] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:24.312621 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:24.319038 30926 raft_consensus.cc:1261] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:03:24.319932 7217 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:24.324458 6439 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:24.325695 30926 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:24.337706 28596 catalog_manager.cc:5484] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 5 to 6, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:24.342311 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0bc741ed064dd7ad8ec70219496c6f with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:24.349825 28874 consensus_queue.cc:237] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:03:24.357802 30926 raft_consensus.cc:1261] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:03:24.358835 28811 consensus_peers.cc:485] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c0bc741ed064dd7ad8ec70219496c6f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:24.359247 7217 consensus_queue.cc:1035] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:24.364724 7217 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:03:24.371776 30926 raft_consensus.cc:2932] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:03:24.379938 28595 catalog_manager.cc:5484] T 7c0bc741ed064dd7ad8ec70219496c6f P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 6 to 7, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:03:24.384403 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7c0bc741ed064dd7ad8ec70219496c6f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:24.387545 7376 tablet_replica.cc:330] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:24.393644 7376 raft_consensus.cc:2229] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:24.394179 7376 raft_consensus.cc:2258] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:24.396188 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.406677 7376 ts_tablet_manager.cc:1875] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:24.414475 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.448345 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:24.514017 29006 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:03:24.516779 6074 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:24.521170 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:24.536480 29136 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:03:24.538684 7241 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:03:24.548967 7376 ts_tablet_manager.cc:1888] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:03:24.549105 7376 log.cc:1196] T 7c0bc741ed064dd7ad8ec70219496c6f P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c0bc741ed064dd7ad8ec70219496c6f
I20240501 14:03:24.549950 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 7c0bc741ed064dd7ad8ec70219496c6f (table rebalance_test_table_5 [id=3b588a9a3cdf4f31bd439f0eafb9fa7a]) successfully deleted
I20240501 14:03:24.556034 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:24.659969 6444 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:24.660231 6444 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:24.661134 6444 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:24.662822 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:24.663729 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:03:24.664011 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:24.664808 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:24.664940 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:24.665452 6444 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:24.666922 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.680398 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:24.689000 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.705550 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:24.719748 30596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 69275.1us
W20240501 14:03:24.723481 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:24.735109 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:24.735950 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:24.744946 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:24.759791 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.763231 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:24.775651 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:24.791847 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.801445 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:24.853629 7452 ts_tablet_manager.cc:902] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:03:24.867543 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:24.896813 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:24.917821 7452 tablet_copy_client.cc:308] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:03:24.918803 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 7c0bc741ed064dd7ad8ec70219496c6f from peer 343a85ba3b514663af5320cd14a9e144 ({username='slave'} at 127.16.36.129:56137)
I20240501 14:03:24.918993 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 7c0bc741ed064dd7ad8ec70219496c6f from peer 343a85ba3b514663af5320cd14a9e144 at {username='slave'} at 127.16.36.129:56137: session id = 343a85ba3b514663af5320cd14a9e144-7c0bc741ed064dd7ad8ec70219496c6f
I20240501 14:03:24.922009 28896 tablet_copy_source_session.cc:205] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:24.923976 7452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0bc741ed064dd7ad8ec70219496c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:24.930995 7452 tablet_copy_client.cc:783] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:24.931443 7452 tablet_copy_client.cc:647] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:24.933768 7452 tablet_copy_client.cc:523] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:24.937981 7452 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:03:25.050187 7452 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:25.050704 7452 tablet_bootstrap.cc:492] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: Bootstrap complete.
I20240501 14:03:25.051023 7452 ts_tablet_manager.cc:1367] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.113s user 0.065s sys 0.040s
I20240501 14:03:25.051918 7452 raft_consensus.cc:348] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:03:25.052153 7452 raft_consensus.cc:729] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:03:25.052486 7452 consensus_queue.cc:260] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:03:25.053200 7452 ts_tablet_manager.cc:1398] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:25.054000 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 343a85ba3b514663af5320cd14a9e144-7c0bc741ed064dd7ad8ec70219496c6f received from {username='slave'} at 127.16.36.129:56137
I20240501 14:03:25.054141 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 343a85ba3b514663af5320cd14a9e144-7c0bc741ed064dd7ad8ec70219496c6f on tablet 7c0bc741ed064dd7ad8ec70219496c6f with peer 343a85ba3b514663af5320cd14a9e144
W20240501 14:03:25.062911 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:25.404495 30793 consensus_queue.cc:237] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:25.407467 29136 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:03:25.408646 7241 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:03:25.427589 7542 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:25.429450 29136 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:25.429751 28743 raft_consensus.cc:1203] T 7c0bc741ed064dd7ad8ec70219496c6f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:03:25.443992 28596 catalog_manager.cc:5484] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:03:25.516320 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:25.524279 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8c435795a04b1b90b6e2ff914676c8 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:25.525946 30793 consensus_queue.cc:237] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
W20240501 14:03:25.529482 30730 consensus_peers.cc:485] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d8c435795a04b1b90b6e2ff914676c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:25.530328 29136 raft_consensus.cc:1261] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:25.531004 7548 consensus_queue.cc:1035] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:25.534580 7241 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:25.542927 29136 raft_consensus.cc:2932] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:25.548389 28601 catalog_manager.cc:5484] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) added. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:03:25.550611 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4d8c435795a04b1b90b6e2ff914676c8 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:25.556978 28987 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:25.577395 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:25.602337 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:25.609822 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:25.628180 7580 tablet_replica.cc:330] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:25.632931 7580 raft_consensus.cc:2229] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:25.633620 7580 raft_consensus.cc:2258] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:25.635418 7580 ts_tablet_manager.cc:1875] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:25.663180 7580 ts_tablet_manager.cc:1888] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:03:25.664554 7580 log.cc:1196] T 4d8c435795a04b1b90b6e2ff914676c8 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/4d8c435795a04b1b90b6e2ff914676c8
I20240501 14:03:25.666951 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 4d8c435795a04b1b90b6e2ff914676c8 (table rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]) successfully deleted
I20240501 14:03:25.718312 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:25.754881 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:25.799898 30596 consensus_peers.cc:462] T 9b7c7b963588436bb778e45da469c7d5 P 26acba62745c48e3ba8180748d55b211 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:25.875667 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:25.889822 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:25.911187 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:25.933486 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:25.952577 30642 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:25.970801 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:25.981410 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:25.990833 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:25.993589 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:26.012140 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:26.014570 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:26.034453 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:26.055177 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:26.066004 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:26.066717 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:26.099998 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:26.102227 7650 ts_tablet_manager.cc:902] T 4d8c435795a04b1b90b6e2ff914676c8 P 26acba62745c48e3ba8180748d55b211: Initiating tablet copy from peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
F20240501 14:03:26.106069 7650 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-014635e01a22420b8bc21e879449749b.queue_time_us, tablet-download-pool-014635e01a22420b8bc21e879449749b.queue_time_us) (tablet-download-pool-014635e01a22420b8bc21e879449749b.run_cpu_time_us, tablet-download-pool-014635e01a22420b8bc21e879449749b.run_cpu_time_us) (tablet-download-pool-014635e01a22420b8bc21e879449749b.run_wall_time_us, tablet-download-pool-014635e01a22420b8bc21e879449749b.run_wall_time_us) (tablet-download-pool-12510f71b6a742ab8cc359243de8119b.queue_time_us, tablet-download-pool-12510f71b6a742ab8cc359243de8119b.queue_time_us) (tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_cpu_time_us, tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_cpu_time_us) (tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_wall_time_us, tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_wall_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.queue_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.queue_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_cpu_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_cpu_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_wall_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_wall_time_us) (tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.queue_time_us, tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.queue_time_us) (tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.queue_time_us, tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.queue_time_us) (tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_cpu_time_us, tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_cpu_time_us) (tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_wall_time_us, tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_wall_time_us) (tablet-download-pool-86bfedcdabe44e7ab135674163935786.queue_time_us, tablet-download-pool-86bfedcdabe44e7ab135674163935786.queue_time_us) (tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_cpu_time_us, tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_cpu_time_us) (tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_wall_time_us, tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_wall_time_us) (tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.queue_time_us, tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.queue_time_us) (tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.run_cpu_time_us, tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.run_cpu_time_us) (tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.run_wall_time_us, tablet-download-pool-9b7c7b963588436bb778e45da469c7d5.run_wall_time_us) (tablet-download-pool-bb712628c7764a138f5425d14b724d9e.queue_time_us, tablet-download-pool-bb712628c7764a138f5425d14b724d9e.queue_time_us) (tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_cpu_time_us, tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_cpu_time_us) (tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_wall_time_us, tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_wall_time_us) (tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.queue_time_us, tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.queue_time_us) (tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_cpu_time_us, tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_cpu_time_us) (tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_wall_time_us, tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_wall_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.queue_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.queue_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_cpu_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_cpu_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_wall_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_wall_time_us) (tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.queue_time_us, tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.queue_time_us) (tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_cpu_time_us, tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_cpu_time_us) (tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_wall_time_us, tablet-download-pool-e821b93aa9b84ff3a968860a5b33b918.run_wall_time_us) (tablet-download-pool-e9bb3eaf3f0e40f8902b9426a845d9b2.queue_time_us, tablet-download-pool-e9bb3eaf3f0e40f8902b9426a845d9b2.queue_time_us) (tablet-download-pool-e9bb3eaf3f0e40f8902b9426a845d9b2.run_cpu_time_us, tablet-download-pool-e9bb3eaf3f0e40f8902b9426a845d9b2.run_cpu_time_us) (tablet-download-pool-e9bb3eaf3f0e40f8902b9426a845d9b2.run_wall_time_us, tablet-download-pool-e9bb3eaf3f0e40f8902b9426a845d9b2.run_wall_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.queue_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.queue_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.run_cpu_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.run_cpu_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.run_wall_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.run_wall_time_us) (tablet-download-pool-f516f01e242d4b8789130201f748f3ea.queue_time_us, tablet-download-pool-f516f01e242d4b8789130201f748f3ea.queue_time_us) (tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_cpu_time_us, tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_cpu_time_us) (tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_wall_time_us, tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_wall_time_us) (tablet-download-pool-fb04ec77da194832aa63d69aba307de3.queue_time_us, tablet-download-pool-fb04ec77da194832aa63d69aba307de3.queue_time_us) (tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_cpu_time_us, tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_cpu_time_us) (tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_wall_time_us, tablet-download-pool-fb04ec77da194832aa63d69aba307de3.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7ff8a92c2dcd google::LogMessage::Fail() at ??:0
W20240501 14:03:26.126757 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:26.131196 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7ff8a92c6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff8a92c27cc google::LogMessage::Flush() at ??:0
I20240501 14:03:26.165647 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:26.189024 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
@ 0x7ff8a92c3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff8ac003742 kudu::TraceMetrics::InternName() at ??:0
W20240501 14:03:26.287624 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
@ 0x7ff8abfcff8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7ff8abfc9cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7ff8bb41c8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7ff8bb434daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7ff8bb55f480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7ff8bb597a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7ff8bb57a981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ff8bb57a70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ff8abfd3b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff8abfd719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff8abfd7019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ff8abfb201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff8af3376db start_thread at ??:0
@ 0x7ff8a817571f clone at ??:0
I20240501 14:03:26.448063 30793 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:26.464491 30926 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:26.465411 7542 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:26.479568 7542 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:26.483539 30927 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:26.490303 28598 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 7 to 9, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
W20240501 14:03:26.506994 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:26.533839 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1b3fec785f43c4a3bacedecd8472dd with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59004
I20240501 14:03:26.540714 30793 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
W20240501 14:03:26.544065 30730 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.545341 30927 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:26.546312 7542 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:26.551371 7542 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:03:26.555548 30927 raft_consensus.cc:2932] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:03:26.557049 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ec1b3fec785f43c4a3bacedecd8472dd with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:26.561704 28598 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 9 to 10, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:03:26.583487 7720 tablet_replica.cc:330] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:03:26.583822 7720 raft_consensus.cc:2229] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:26.584046 7720 raft_consensus.cc:2258] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:03:26.585090 7720 ts_tablet_manager.cc:1875] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:26.636546 7720 ts_tablet_manager.cc:1888] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:03:26.636683 7720 log.cc:1196] T ec1b3fec785f43c4a3bacedecd8472dd P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/ec1b3fec785f43c4a3bacedecd8472dd
I20240501 14:03:26.639456 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet ec1b3fec785f43c4a3bacedecd8472dd (table rebalance_test_table_7 [id=771fb3db7ee841fd8e33c05cb3a9b82e]) successfully deleted
W20240501 14:03:26.707396 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:26.744836 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:26.792438 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:26.861992 29073 connection.cc:535] client connection to 127.16.36.133:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:26.862246 28811 connection.cc:535] client connection to 127.16.36.133:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:26.863159 28680 connection.cc:535] client connection to 127.16.36.133:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:26.863734 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.863772 30730 connection.cc:535] client connection to 127.16.36.133:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:26.863839 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.863899 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.864593 28942 connection.cc:535] client connection to 127.16.36.133:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:03:26.866997 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.867105 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.867305 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:26.867389 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.868963 6444 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:26.869118 6444 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:26.869830 6444 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:03:26.870149 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.870273 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.870344 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:26.871718 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:26.871836 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:26.872411 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:03:26.872509 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:26.872997 6444 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:03:26.873248 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.873373 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.873443 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:26.873499 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:03:27.029439 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:27.160090 7850 ts_tablet_manager.cc:902] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Initiating tablet copy from peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:27.168051 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.199586 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.216812 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.244227 7850 tablet_copy_client.cc:308] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.134:44241
I20240501 14:03:27.245683 30816 tablet_copy_service.cc:140] P 8fc5d8222c164a679e7a99966da1eaaf: Received BeginTabletCopySession request for tablet ec1b3fec785f43c4a3bacedecd8472dd from peer 343a85ba3b514663af5320cd14a9e144 ({username='slave'} at 127.16.36.129:35239)
I20240501 14:03:27.245887 30816 tablet_copy_service.cc:161] P 8fc5d8222c164a679e7a99966da1eaaf: Beginning new tablet copy session on tablet ec1b3fec785f43c4a3bacedecd8472dd from peer 343a85ba3b514663af5320cd14a9e144 at {username='slave'} at 127.16.36.129:35239: session id = 343a85ba3b514663af5320cd14a9e144-ec1b3fec785f43c4a3bacedecd8472dd
I20240501 14:03:27.249747 30816 tablet_copy_source_session.cc:205] T ec1b3fec785f43c4a3bacedecd8472dd P 8fc5d8222c164a679e7a99966da1eaaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:27.251713 7850 ts_tablet_manager.cc:1875] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:27.258773 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.276250 7762 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:03:27.276477 7762 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:27.277226 7762 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:03:27.281003 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:27.281431 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:27.281548 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:27.282146 7762 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:27.318711 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.349453 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.360026 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.389752 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.396981 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:27.371275 7850 ts_tablet_manager.cc:1888] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:03:27.411556 7850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1b3fec785f43c4a3bacedecd8472dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:27.444653 7850 tablet_copy_client.cc:783] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:27.444934 7850 tablet_copy_client.cc:647] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:27.447896 7850 tablet_copy_client.cc:523] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:27.459877 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.463392 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.475026 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:27.476193 7850 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
W20240501 14:03:27.491176 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:27.545461 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.552387 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.552716 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.567137 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.606058 29118 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.622140 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:27.627089 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.635869 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.702083 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.715823 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:27.724534 30775 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:27.728113 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:27.847337 7850 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:27.847828 7850 tablet_bootstrap.cc:492] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Bootstrap complete.
I20240501 14:03:27.848132 7850 ts_tablet_manager.cc:1367] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.372s user 0.060s sys 0.001s
I20240501 14:03:27.850852 7850 raft_consensus.cc:348] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:03:27.851126 7850 raft_consensus.cc:729] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:03:27.851866 7850 consensus_queue.cc:260] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:03:27.853534 7850 ts_tablet_manager.cc:1398] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240501 14:03:27.854344 30816 tablet_copy_service.cc:342] P 8fc5d8222c164a679e7a99966da1eaaf: Request end of tablet copy session 343a85ba3b514663af5320cd14a9e144-ec1b3fec785f43c4a3bacedecd8472dd received from {username='slave'} at 127.16.36.129:35239
I20240501 14:03:27.854488 30816 tablet_copy_service.cc:433] P 8fc5d8222c164a679e7a99966da1eaaf: ending tablet copy session 343a85ba3b514663af5320cd14a9e144-ec1b3fec785f43c4a3bacedecd8472dd on tablet ec1b3fec785f43c4a3bacedecd8472dd with peer 343a85ba3b514663af5320cd14a9e144
I20240501 14:03:27.944792 29136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177"
dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33896
I20240501 14:03:27.945080 29136 raft_consensus.cc:595] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Received request to transfer leadership
I20240501 14:03:28.027366 7955 raft_consensus.cc:979] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: : Instructing follower 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:28.028455 7955 raft_consensus.cc:1067] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Signalling peer 8fc5d8222c164a679e7a99966da1eaaf to start an election
I20240501 14:03:28.033566 30793 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177"
dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf"
from {username='slave'} at 127.16.36.132:59495
I20240501 14:03:28.033857 30793 raft_consensus.cc:482] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:03:28.034036 30793 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:28.037876 30793 raft_consensus.cc:504] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
W20240501 14:03:28.040119 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:28.041363 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:03:28.042788 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "8fc5d8222c164a679e7a99966da1eaaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a0e901103604414c829b40edf9f24c28"
I20240501 14:03:28.043038 30927 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:28.048394 28743 raft_consensus.cc:1203] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 14:03:28.054935 30927 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:28.055644 30730 leader_election.cc:302] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fc5d8222c164a679e7a99966da1eaaf, a0e901103604414c829b40edf9f24c28; no voters:
I20240501 14:03:28.056283 7542 raft_consensus.cc:2781] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:03:28.058113 30793 leader_election.cc:288] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [CANDIDATE]: Term 2 election: Requested vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:03:28.063084 7542 raft_consensus.cc:686] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Becoming Leader. State: Replica: 8fc5d8222c164a679e7a99966da1eaaf, State: Running, Role: LEADER
I20240501 14:03:28.063966 7542 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:28.074096 29136 raft_consensus.cc:3032] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:03:28.074303 29136 raft_consensus.cc:729] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Running, Role: LEADER
I20240501 14:03:28.076622 29136 consensus_queue.cc:260] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:28.077301 29136 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:03:28.082609 29136 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fc5d8222c164a679e7a99966da1eaaf in term 2.
I20240501 14:03:28.088910 28598 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: term changed from 1 to 2, leader changed from 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) to 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134). New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
W20240501 14:03:28.393751 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:28.393805 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:28.402772 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:28.415525 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:28.420832 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:28.440294 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:28.441222 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:28.444516 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:28.448999 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:28.463395 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:28.464449 29136 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:28.465039 7967 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:03:28.486886 30927 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:03:28.487807 7967 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:03:28.504380 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:28.513525 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:28.555243 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:28.564952 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:28.566957 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:28.571612 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:28.573055 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:28.583636 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:28.800959 31113 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task1g_oHS/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:02:28.088404 31119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:28.088900 31119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:28.102974 31119 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:31.982002 31119 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:02:33.217160 31777 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 78578.4us
W20240501 14:02:33.754897 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.755307 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:33.755488 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.755631 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:33.755781 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.759133 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.759320 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.759423 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:33.759577 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:33.759783 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:33.759904 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.760008 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:33.760159 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.760272 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:34.368631 31119 rebalancer_tool.cc:1074] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:34.833546 31119 rebalancer_tool.cc:1084] tablet f516f01e242d4b8789130201f748f3ea: '7e7bc120bd74479ab378f779778fd1e8' -> '8fc5d8222c164a679e7a99966da1eaaf' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:02:34.833715 31119 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:36.968884 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.969321 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.969499 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.969864 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.969990 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.970132 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.970247 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.970396 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.970484 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.970592 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.979061 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.979306 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.979409 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.979502 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.979593 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.979698 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:37.180859 31119 rebalancer_tool.cc:1074] tablet 2020fe37059a4e74bb45190398e4e177: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:02:37.997331 31119 rebalancer_tool.cc:1074] tablet fb04ec77da194832aa63d69aba307de3: 'cd3c04ebb6c84b15a1b3adc748737094' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:38.580272 31119 rebalancer_tool.cc:1074] tablet 135da52054584a27959efdd9126cfcef: '3a3972b170c54344a4ba61f176f837ba' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:39.143046 31119 rebalancer_tool.cc:1074] tablet ec1b3fec785f43c4a3bacedecd8472dd: '343a85ba3b514663af5320cd14a9e144' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:39.555075 31119 rebalancer_tool.cc:1074] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:39.662739 31119 rebalancer_tool.cc:1074] tablet 60b3db79ff754e8bbe9e5865f7104312: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:02:40.017840 31119 rebalancer_tool.cc:1074] tablet 014635e01a22420b8bc21e879449749b: '3a3972b170c54344a4ba61f176f837ba' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:40.680223 31119 rebalancer_tool.cc:1074] tablet e0bf08cbb6d347d783b1fa7762c802df: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:41.446699 31119 rebalancer_tool.cc:1074] tablet 2998b60071cd408aa1cd06fb57d76a8f: '7e7bc120bd74479ab378f779778fd1e8' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:41.770073 31119 rebalancer_tool.cc:1074] tablet 12510f71b6a742ab8cc359243de8119b: 'cd3c04ebb6c84b15a1b3adc748737094' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:42.202136 31119 rebalancer_tool.cc:1074] tablet e9bb3eaf3f0e40f8902b9426a845d9b2: '3a3972b170c54344a4ba61f176f837ba' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:42.612977 31119 rebalancer_tool.cc:1074] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:43.311867 31119 rebalancer_tool.cc:1074] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:43.692499 31119 rebalancer_tool.cc:1074] tablet de161b10c84f4bcc85f8cf7b199f99db: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:44.031757 31119 rebalancer_tool.cc:1074] tablet f4402de0b48041c383568feb9365147e: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:44.683915 31119 rebalancer_tool.cc:1074] tablet 1fdedd39e12a49659d989f37b3287d87: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:44.823784 31119 rebalancer_tool.cc:1084] tablet 39f1ab3a94db41b08ac90bba52418a93: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move ignored: Illegal state: tablet 39f1ab3a94db41b08ac90bba52418a93: replica 8fc5d8222c164a679e7a99966da1eaaf is already present
I20240501 14:02:44.823995 31119 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:46.931761 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server a0e901103604414c829b40edf9f24c28 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:46.932142 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:46.932364 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:46.932503 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:46.932674 31119 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:47.360863 31119 rebalancer_tool.cc:1074] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '826ccf8358b6453d88bcb132290b82a5' -> '' move scheduled
I20240501 14:02:47.442924 31119 rebalancer_tool.cc:1074] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:02:47.526966 31119 rebalancer_tool.cc:1074] tablet 39f1ab3a94db41b08ac90bba52418a93: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:02:59.834384 31119 rebalancer_tool.cc:1136] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:02:59.927073 31119 rebalancer_tool.cc:1136] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:07.559983 31119 rebalancer_tool.cc:1136] tablet 014635e01a22420b8bc21e879449749b: '3a3972b170c54344a4ba61f176f837ba' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:14.046449 31119 rebalancer_tool.cc:1136] tablet 2998b60071cd408aa1cd06fb57d76a8f: '7e7bc120bd74479ab378f779778fd1e8' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:14.108373 31119 rebalancer_tool.cc:1136] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:14.211017 31119 rebalancer_tool.cc:1136] tablet f4402de0b48041c383568feb9365147e: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:21.373473 31119 rebalancer_tool.cc:1136] tablet fb04ec77da194832aa63d69aba307de3: 'cd3c04ebb6c84b15a1b3adc748737094' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:21.428997 31119 rebalancer_tool.cc:1136] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:21.471760 31119 rebalancer_tool.cc:1136] tablet de161b10c84f4bcc85f8cf7b199f99db: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:21.707993 31119 rebalancer_tool.cc:1074] tablet 4d8c435795a04b1b90b6e2ff914676c8: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:21.791499 31119 rebalancer_tool.cc:1136] tablet 39f1ab3a94db41b08ac90bba52418a93: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:21.871680 31119 rebalancer_tool.cc:1136] tablet ec1b3fec785f43c4a3bacedecd8472dd: '343a85ba3b514663af5320cd14a9e144' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:21.942355 31119 rebalancer_tool.cc:1136] tablet 60b3db79ff754e8bbe9e5865f7104312: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:25.824009 31119 rebalancer_tool.cc:1136] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:26.031924 31119 rebalancer_tool.cc:1136] tablet 2020fe37059a4e74bb45190398e4e177: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:26.208694 31119 rebalancer_tool.cc:1136] tablet 1fdedd39e12a49659d989f37b3287d87: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:26.369518 31119 rebalancer_tool.cc:1136] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '826ccf8358b6453d88bcb132290b82a5' -> '' move completed: OK
I20240501 14:03:26.453639 31119 rebalancer_tool.cc:1136] tablet e0bf08cbb6d347d783b1fa7762c802df: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move completed: Incomplete: tablet e0bf08cbb6d347d783b1fa7762c802df, TS 343a85ba3b514663af5320cd14a9e144 -> TS 826ccf8358b6453d88bcb132290b82a5 move failed, target replica disappeared
I20240501 14:03:26.639851 31119 rebalancer_tool.cc:1136] tablet 12510f71b6a742ab8cc359243de8119b: 'cd3c04ebb6c84b15a1b3adc748737094' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:26.772478 31119 rebalancer_tool.cc:1136] tablet 135da52054584a27959efdd9126cfcef: '3a3972b170c54344a4ba61f176f837ba' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:26.826840 31119 rebalancer_tool.cc:1136] tablet e9bb3eaf3f0e40f8902b9426a845d9b2: '3a3972b170c54344a4ba61f176f837ba' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:26.848515 31119 rebalancer_tool.cc:1084] tablet e8b9db597e654ed48b679abf58b8a2c8: '343a85ba3b514663af5320cd14a9e144' -> '' move ignored: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:26.848672 31119 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): unacceptable health status UNAVAILABLE
W20240501 14:03:28.810653 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:28.850121 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:28.850984 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:28.870729 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:28.899895 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:28.917129 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:28.922766 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:28.959810 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:28.979835 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:03:28.982431 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:28.994817 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.001410 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:03:29.005203 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:29.008030 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:29.014912 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:29.015570 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:29.084090 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240501 14:03:29.085644 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.118028 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:03:29.125078 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.207065 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:29.207895 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:29.211540 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:29.226155 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:29.245685 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:29.247822 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:29.256843 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:29.261873 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:03:29.284817 7913 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:03:29.285039 7913 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:29.285733 7913 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:03:29.288154 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:03:29.289934 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.294795 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:29.295673 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:29.295888 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:29.296319 7913 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:03:29.322304 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:29.324693 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:29.345108 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:29.404707 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.419108 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.436839 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.442471 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:03:29.505404 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240501 14:03:29.575124 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:29.608814 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:29.632098 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:29.650072 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:29.669194 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:29.726785 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:29.736258 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:29.739732 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:29.758172 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:29.778223 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:29.838441 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:29.865252 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:29.889714 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.029498 31110 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task1g_oHS/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:02:27.668221 31115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:27.668733 31115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:27.681921 31115 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:30.708585 31115 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:02:32.744326 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.747632 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:32.748171 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.748502 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:32.755045 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:32.758810 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:32.759259 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.759594 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:32.760035 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:32.760470 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:32.760763 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.761091 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:32.761411 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:32.761770 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:32.762089 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
I20240501 14:02:33.490700 31115 rebalancer_tool.cc:1074] tablet 60b3db79ff754e8bbe9e5865f7104312: '343a85ba3b514663af5320cd14a9e144' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:33.950258 31115 rebalancer_tool.cc:1084] tablet 2020fe37059a4e74bb45190398e4e177: '7e7bc120bd74479ab378f779778fd1e8' -> '26acba62745c48e3ba8180748d55b211' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:02:33.952759 31115 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:36.177592 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.177785 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.178045 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.178234 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.178395 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.178495 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.178679 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.178786 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.179035 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.179147 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.179253 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.179371 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.179464 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.179572 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
I20240501 14:02:36.425628 31115 rebalancer_tool.cc:1074] tablet 2020fe37059a4e74bb45190398e4e177: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:02:36.935127 31115 rebalancer_tool.cc:1074] tablet 111ba172d7e44054872c8fc8cac8ba88: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:37.334401 31115 rebalancer_tool.cc:1074] tablet 39f1ab3a94db41b08ac90bba52418a93: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:37.640775 31115 rebalancer_tool.cc:1074] tablet 12510f71b6a742ab8cc359243de8119b: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:38.489562 31115 rebalancer_tool.cc:1074] tablet 2998b60071cd408aa1cd06fb57d76a8f: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:38.658380 31115 rebalancer_tool.cc:1074] tablet 86bfedcdabe44e7ab135674163935786: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:02:39.354069 31115 rebalancer_tool.cc:1074] tablet f4402de0b48041c383568feb9365147e: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:39.485502 31115 rebalancer_tool.cc:1084] tablet 135da52054584a27959efdd9126cfcef: '3a3972b170c54344a4ba61f176f837ba' -> '826ccf8358b6453d88bcb132290b82a5' move ignored: Illegal state: tablet 135da52054584a27959efdd9126cfcef: replica 826ccf8358b6453d88bcb132290b82a5 is already present
I20240501 14:02:39.485625 31115 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:41.514031 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:41.514312 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:41.514432 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:41.514590 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:41.514743 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:41.514883 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:41.515040 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:41.516616 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:41.516789 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:41.516898 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:41.517009 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:41.517127 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
I20240501 14:02:42.228616 31115 rebalancer_tool.cc:1074] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:42.387800 31115 rebalancer_tool.cc:1074] tablet f516f01e242d4b8789130201f748f3ea: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:02:42.999078 31115 rebalancer_tool.cc:1074] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:43.095402 31115 rebalancer_tool.cc:1074] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:02:43.544320 31115 rebalancer_tool.cc:1074] tablet 9b7c7b963588436bb778e45da469c7d5: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:43.669616 31115 rebalancer_tool.cc:1074] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:02:44.357179 31115 rebalancer_tool.cc:1074] tablet 4d8c435795a04b1b90b6e2ff914676c8: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:44.522428 31115 rebalancer_tool.cc:1084] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move ignored: Illegal state: tablet 63b7f2a8d8a84faf86c4de16b3cc6663: replica 8fc5d8222c164a679e7a99966da1eaaf is already present
I20240501 14:02:44.522645 31115 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:46.412865 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:46.413076 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:46.413192 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:46.413345 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:46.413483 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:46.413616 31115 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:55.584755 31115 rebalancer_tool.cc:1136] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:01.180390 31115 rebalancer_tool.cc:1136] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:01.390992 31115 rebalancer_tool.cc:1136] tablet 12510f71b6a742ab8cc359243de8119b: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:06.573236 31115 rebalancer_tool.cc:1136] tablet 39f1ab3a94db41b08ac90bba52418a93: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
W20240501 14:03:07.895864 4476 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 103611us
I20240501 14:03:09.317080 31115 rebalancer_tool.cc:1136] tablet 2020fe37059a4e74bb45190398e4e177: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:11.789064 31115 rebalancer_tool.cc:1074] tablet e0bf08cbb6d347d783b1fa7762c802df: '826ccf8358b6453d88bcb132290b82a5' -> '' move scheduled
I20240501 14:03:11.916023 31115 rebalancer_tool.cc:1136] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:12.005481 31115 rebalancer_tool.cc:1136] tablet 9b7c7b963588436bb778e45da469c7d5: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:12.147099 31115 rebalancer_tool.cc:1136] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:12.510959 31115 rebalancer_tool.cc:1136] tablet f516f01e242d4b8789130201f748f3ea: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:12.680672 31115 rebalancer_tool.cc:1136] tablet 86bfedcdabe44e7ab135674163935786: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:12.878474 31115 rebalancer_tool.cc:1136] tablet 60b3db79ff754e8bbe9e5865f7104312: '343a85ba3b514663af5320cd14a9e144' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:13.012147 31115 rebalancer_tool.cc:1136] tablet 111ba172d7e44054872c8fc8cac8ba88: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:13.099714 31115 rebalancer_tool.cc:1136] tablet 4d8c435795a04b1b90b6e2ff914676c8: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:13.214926 31115 rebalancer_tool.cc:1136] tablet e0bf08cbb6d347d783b1fa7762c802df: '826ccf8358b6453d88bcb132290b82a5' -> '' move completed: OK
I20240501 14:03:13.541587 31115 rebalancer_tool.cc:1136] tablet f4402de0b48041c383568feb9365147e: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:13.656780 31115 rebalancer_tool.cc:1136] tablet 2998b60071cd408aa1cd06fb57d76a8f: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:13.725389 31115 rebalancer_tool.cc:1074] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:13.792369 31115 rebalancer_tool.cc:1074] tablet de161b10c84f4bcc85f8cf7b199f99db: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:03:13.892104 31115 rebalancer_tool.cc:1074] tablet 135da52054584a27959efdd9126cfcef: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:14.022720 31115 rebalancer_tool.cc:1074] tablet ec1b3fec785f43c4a3bacedecd8472dd: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:14.144011 31115 rebalancer_tool.cc:1074] tablet e8b9db597e654ed48b679abf58b8a2c8: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:17.349521 31115 rebalancer_tool.cc:1136] tablet de161b10c84f4bcc85f8cf7b199f99db: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
W20240501 14:03:18.933032 6328 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:03:18.384604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.136:36399 (local address 127.0.0.1:42626)
0501 14:03:18.384886 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0501 14:03:18.384907 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 14:03:18.385044 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:03:18.931429 (+546385us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121}
I20240501 14:03:23.439240 31115 rebalancer_tool.cc:1074] tablet 9b473e10a8ad4bd58bb44aabd9656e08: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:25.035807 31115 rebalancer_tool.cc:1123] tablet e8b9db597e654ed48b679abf58b8a2c8: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:28.852810 31115 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): unacceptable health status UNAVAILABLE
I20240501 14:03:30.099574 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:30.124522 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.129114 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:30.136729 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:30.156646 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.162652 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:30.221781 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:30.471109 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:30.475019 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.502449 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:30.516970 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.561560 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.561975 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:30.619421 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:30.639439 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:30.645962 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:30.665535 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:30.677673 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.679172 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.707168 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:30.714262 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:30.895028 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:30.906505 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:30.932125 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.932257 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.938263 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:30.938441 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:30.975220 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:31.041256 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:31.069628 30793 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:03:31.071877 30926 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:03:31.072435 8142 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:03:31.075038 8311 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:31.075809 30926 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:31.081488 28595 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 7 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) evicted. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:03:31.092319 30793 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:03:31.092564 29118 tablet_service.cc:1514] Processing DeleteTablet for tablet 2020fe37059a4e74bb45190398e4e177 with delete_type TABLET_DATA_TOMBSTONED (TS 3a3972b170c54344a4ba61f176f837ba not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50834
I20240501 14:03:31.093317 8397 tablet_replica.cc:330] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: stopping tablet replica
I20240501 14:03:31.093510 8397 raft_consensus.cc:2229] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:03:31.094272 8397 raft_consensus.cc:2258] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 14:03:31.095713 30728 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Couldn't send request to peer cd3c04ebb6c84b15a1b3adc748737094. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:03:31.095737 30926 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:03:31.096561 8142 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:03:31.098136 8397 ts_tablet_manager.cc:1875] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:03:31.100091 8142 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:03:31.100499 30926 raft_consensus.cc:2932] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:03:31.106236 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2020fe37059a4e74bb45190398e4e177 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:03:31.108286 28595 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 9 to 10, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
W20240501 14:03:31.112701 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:31.111768 8397 ts_tablet_manager.cc:1888] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:03:31.113198 8397 log.cc:1196] T 2020fe37059a4e74bb45190398e4e177 P 3a3972b170c54344a4ba61f176f837ba: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/wals/2020fe37059a4e74bb45190398e4e177
I20240501 14:03:31.114277 28588 catalog_manager.cc:4830] TS 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): tablet 2020fe37059a4e74bb45190398e4e177 (table rebalance_test_table_1 [id=434d653274bc4aa7938133d6f5481332]) successfully deleted
W20240501 14:03:31.155453 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:31.198011 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:31.208039 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:31.249186 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:31.263991 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:31.268393 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:31.281841 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:31.290963 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:31.307638 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:31.311575 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:31.332979 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:31.366544 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:31.366564 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:31.450949 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:31.463219 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:31.467437 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:31.467553 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:31.476627 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:31.493798 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:31.501278 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:31.510202 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:31.532034 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:31.541827 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:31.573647 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:31.654374 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:31.660018 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:31.688933 8500 ts_tablet_manager.cc:902] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Initiating tablet copy from peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:31.700593 8500 tablet_copy_client.cc:308] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.134:44241
I20240501 14:03:31.701762 30816 tablet_copy_service.cc:140] P 8fc5d8222c164a679e7a99966da1eaaf: Received BeginTabletCopySession request for tablet 2020fe37059a4e74bb45190398e4e177 from peer cd3c04ebb6c84b15a1b3adc748737094 ({username='slave'} at 127.16.36.130:55461)
I20240501 14:03:31.701954 30816 tablet_copy_service.cc:161] P 8fc5d8222c164a679e7a99966da1eaaf: Beginning new tablet copy session on tablet 2020fe37059a4e74bb45190398e4e177 from peer cd3c04ebb6c84b15a1b3adc748737094 at {username='slave'} at 127.16.36.130:55461: session id = cd3c04ebb6c84b15a1b3adc748737094-2020fe37059a4e74bb45190398e4e177
I20240501 14:03:31.704891 30816 tablet_copy_source_session.cc:205] T 2020fe37059a4e74bb45190398e4e177 P 8fc5d8222c164a679e7a99966da1eaaf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:03:31.707405 8500 ts_tablet_manager.cc:1875] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:03:31.717226 8500 ts_tablet_manager.cc:1888] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:03:31.717751 8500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2020fe37059a4e74bb45190398e4e177. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:03:31.718113 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:31.721884 8500 tablet_copy_client.cc:783] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 0 data blocks...
I20240501 14:03:31.722270 8500 tablet_copy_client.cc:647] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 1 WAL segments...
I20240501 14:03:31.724655 8500 tablet_copy_client.cc:523] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:03:31.725729 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:31.729656 8500 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:03:31.760607 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:31.762436 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:31.793349 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:31.799147 8500 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:03:31.799829 8500 tablet_bootstrap.cc:492] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap complete.
I20240501 14:03:31.800304 8500 ts_tablet_manager.cc:1367] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.071s user 0.051s sys 0.011s
I20240501 14:03:31.801393 8500 raft_consensus.cc:348] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:03:31.801735 8500 raft_consensus.cc:729] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:03:31.802093 8500 consensus_queue.cc:260] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:03:31.810093 8500 ts_tablet_manager.cc:1398] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:03:31.811468 30816 tablet_copy_service.cc:342] P 8fc5d8222c164a679e7a99966da1eaaf: Request end of tablet copy session cd3c04ebb6c84b15a1b3adc748737094-2020fe37059a4e74bb45190398e4e177 received from {username='slave'} at 127.16.36.130:55461
I20240501 14:03:31.811652 30816 tablet_copy_service.cc:433] P 8fc5d8222c164a679e7a99966da1eaaf: ending tablet copy session cd3c04ebb6c84b15a1b3adc748737094-2020fe37059a4e74bb45190398e4e177 on tablet 2020fe37059a4e74bb45190398e4e177 with peer cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:03:31.837244 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:31.838006 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:31.862702 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:31.870694 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:31.888092 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:03:31.937536 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:32.013830 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:32.018183 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:32.044116 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:32.044886 28874 raft_consensus.cc:1203] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240501 14:03:32.074543 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:32.080993 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.081171 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.092872 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:32.123068 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:32.160348 8160 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:03:32.160562 8160 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:32.161306 8160 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:03:32.164405 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:32.164722 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:32.164855 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:32.165251 8160 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:03:32.190083 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:03:32.267259 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:32.268078 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:32.289446 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.311334 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.328011 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:32.340207 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:32.352061 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:32.525879 7956 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:32.526072 7956 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:32.526839 7956 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:32.528267 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:32.528937 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:03:32.530014 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:32.530133 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:32.530491 7956 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:32.621425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:32.625741 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:32.652594 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.681277 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.687090 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.699762 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:32.820786 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:32.827561 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:32.845772 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:32.847890 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.857652 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:32.869807 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:32.895107 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:32.935838 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 59839.7us
W20240501 14:03:32.935972 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:33.126286 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:33.134146 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:33.152931 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:33.158555 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:33.165004 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:33.176877 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:33.342325 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:33.353667 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:33.391297 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:33.412303 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:33.419278 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:33.456061 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:33.466445 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:33.487655 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:33.592067 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:33.597643 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:33.646970 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:33.663012 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:33.672474 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:33.680269 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:33.685086 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:33.698189 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:33.738937 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:33.764679 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:33.803920 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:33.835237 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:33.918385 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:33.919631 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:33.934653 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:33.939298 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:33.959483 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:33.975828 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:33.998721 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:34.002996 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:03:34.004264 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:34.004244 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:34.005430 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.024063 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:03:34.025130 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.030259 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:34.030606 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:34.102759 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:03:34.103751 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.145188 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.184370 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.191390 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:34.217896 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:34.231184 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:34.232697 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:34.280249 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:34.280815 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:34.291237 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:34.303306 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.332125 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:34.343470 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:03:34.354545 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:34.455760 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.456979 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:03:34.457912 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:34.555716 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:34.572615 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:34.584843 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:34.594094 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:34.618201 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:34.618281 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:34.717097 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:03:34.749385 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:03:34.758131 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:34.760537 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:34.797732 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:34.808368 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:34.813481 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:34.840234 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:03:34.851217 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:34.858563 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.065302 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:35.071578 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:35.094208 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.110697 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:35.111603 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.119637 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.319887 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:35.321475 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:35.334976 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.338759 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.343616 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:35.362072 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:35.439678 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:35.541011 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:35.573666 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:35.573921 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.581383 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:35.584456 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.602790 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:35.643944 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.887451 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.923285 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:35.944380 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:35.944636 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:35.946671 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:35.957981 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:35.958748 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:35.973224 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:36.048781 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:36.055606 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:36.066913 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.073411 7956 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:36.073621 7956 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:36.074548 7956 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:36.075525 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:36.076596 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:03:36.077555 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:36.077667 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:36.077912 7956 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:36.080727 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.088419 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:36.109311 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:36.129613 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:36.140167 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:36.143508 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:36.232713 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:36.269018 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:36.273978 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:36.326586 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:36.350513 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:36.354255 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:36.369349 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.369982 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:36.387646 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:36.389072 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:36.410705 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:36.420425 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:36.421841 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:36.485463 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:36.494652 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:36.509528 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:36.552587 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:36.578481 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:36.583542 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:36.594039 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:03:36.597965 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:36.603274 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:36.605243 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.605603 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.630885 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:36.630867 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:36.696187 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:36.854240 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.866847 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:36.866940 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:36.872153 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.883996 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:36.899601 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:36.922196 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:36.947623 9026 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:03:36.947804 9026 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:36.948558 9026 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:03:36.950001 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:36.950695 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:36.951203 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:36.951311 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:36.951524 9026 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:03:37.123401 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:37.213130 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:37.233243 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:37.274014 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:37.301502 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:37.310114 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:37.328882 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:37.344955 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:37.398551 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:37.440039 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:37.480588 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:37.484473 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:37.499118 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:03:37.514276 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:37.541934 28940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 44199.8us
I20240501 14:03:37.550328 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:37.571763 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:37.597143 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:37.784685 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:37.788679 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:37.792564 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:37.793347 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:37.793464 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:37.800621 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:38.013701 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:38.050468 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:38.059909 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:38.064232 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:38.071652 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:38.084969 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:38.096379 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:38.115521 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:38.291756 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:38.297225 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:38.304646 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:38.322916 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:38.328490 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:38.343235 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:38.438894 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:38.489763 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:38.578066 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:38.586695 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:38.588449 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:38.593148 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:38.605499 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:38.634496 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:38.647219 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:38.654655 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:38.682271 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:38.684300 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:38.687495 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:38.775557 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:38.776245 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:38.800676 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:38.819821 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:38.826069 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:38.831749 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:38.834748 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:38.835454 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:38.839010 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:38.841840 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:38.878695 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:38.929443 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:38.953066 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:38.971625 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:38.983003 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:38.998785 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:39.060065 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.060534 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:39.067399 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:39.086898 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.097055 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:39.123924 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:39.128587 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:03:39.152925 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:39.154456 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:03:39.244901 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:39.296356 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 14:03:39.305951 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:39.308152 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:39.319269 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.322058 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.335162 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.341606 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:39.372504 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:39.407301 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:03:39.440433 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:39.509588 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:03:39.523387 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:39.578320 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:39.580256 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:03:39.598361 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:39.599843 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.600881 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.609776 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:39.654687 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.657871 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:39.751207 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:39.798451 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:03:39.892187 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:39.905369 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:39.914896 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.931231 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.931710 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:39.940763 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:40.091465 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:40.101379 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:40.126891 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:40.142953 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:40.161130 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:40.174893 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:03:40.180613 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:40.195479 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:40.282476 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:40.301405 7956 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:40.301566 7956 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:40.302142 7956 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:40.303587 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:40.304395 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:03:40.304620 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:40.304738 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:40.305056 7956 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:40.437423 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:40.439286 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:40.469800 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:40.491168 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:40.519117 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:40.529687 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:40.559391 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:40.570391 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:40.614830 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:40.630443 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:40.656260 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:40.656164 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 53389.5us
I20240501 14:03:40.678944 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:40.697224 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:40.893069 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:40.907069 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:40.928119 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:41.004940 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:41.020895 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:41.024385 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:41.089160 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:41.109421 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:41.114976 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.117201 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.131233 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.134290 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:41.159561 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:41.167682 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:41.170691 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:41.194981 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:41.226275 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:41.282521 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:41.296227 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:41.317145 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:41.332171 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.334601 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.345391 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:41.360898 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:41.379801 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:41.443130 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:41.478021 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:41.498266 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:41.506559 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:41.520002 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:41.526755 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.541600 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:41.542958 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:41.553267 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:41.557413 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:41.560451 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.561645 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:41.772972 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:41.812476 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:41.834105 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:41.846117 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.867735 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:41.890965 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:41.902838 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.927691 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.946204 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.948637 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:41.952848 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:41.984318 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:41.998886 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:42.014730 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:42.038491 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:42.083778 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:42.099112 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:42.345535 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:03:42.510787 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:03:42.563481 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:42.568557 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:42.577947 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:42.598811 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:42.636520 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:42.691947 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:42.693118 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:42.706143 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:42.713133 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:42.721930 9748 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:03:42.722143 9748 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:42.722901 9748 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
I20240501 14:03:42.724105 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:42.726174 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:42.728842 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:42.754741 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:42.754981 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:42.755318 9748 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:42.756035 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:42.774168 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:42.832226 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:42.981055 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:42.986853 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:42.997066 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:43.012777 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.033632 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:43.047931 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:43.143779 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:43.196329 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:43.265153 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:43.273761 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:43.329221 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:43.353193 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.369392 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.369536 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:43.406845 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:43.420953 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:43.434675 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:43.463263 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:43.487141 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:43.494729 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:43.495565 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.499786 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:43.503520 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.522459 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:43.587757 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:43.608933 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:43.613478 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:43.643225 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:43.644696 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:43.689805 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:43.777199 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:43.809713 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:43.887136 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:43.893432 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:43.894147 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:43.912882 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.931717 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.944188 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:43.946039 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:44.043678 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:44.058331 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:44.118052 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:44.119081 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:44.127804 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.132475 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:44.137110 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:03:44.144212 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:44.147789 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.163517 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:44.183696 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:03:44.211117 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:44.244189 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:44.345877 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:44.357141 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:03:44.375471 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:44.420115 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:44.428172 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:44.430887 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.457826 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.472667 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:44.509598 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:44.544903 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:03:44.552443 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:44.588192 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:44.626675 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:44.627480 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:44.634392 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:44.653333 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:03:44.667840 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.709743 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:44.726929 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:44.832319 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:44.928159 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:44.929863 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:44.940080 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.952039 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.970992 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:44.976480 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:45.015195 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:03:45.158882 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:45.171656 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.180306 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:45.192857 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:45.207177 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.234835 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:45.244113 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:03:45.277710 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:03:45.415438 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:45.427345 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:45.439148 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:45.444386 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:45.446555 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.456075 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:45.486095 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.526857 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:45.684546 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:45.702930 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:45.714119 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.741132 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:45.754491 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:45.759605 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:45.802516 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:45.823843 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.829728 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:45.860723 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:45.916198 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:45.924829 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:45.953637 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:45.954346 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.966313 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:45.983086 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:46.050076 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:46.113687 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:46.124015 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:46.154054 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:46.166949 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:46.187534 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:46.193717 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:46.211592 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:46.223412 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:46.225505 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:46.229166 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:46.260074 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:46.272420 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:46.312103 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:46.353750 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 50049us
W20240501 14:03:46.393086 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:46.402169 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:46.425477 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:46.480774 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:46.481644 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:46.515792 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:46.523547 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:46.528434 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:46.541639 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:46.559433 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:46.747524 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:46.752087 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:46.762192 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:46.771535 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:46.774591 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:46.790681 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:46.804214 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:46.811154 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:46.811899 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:46.837193 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:46.858116 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:03:46.877207 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:46.963533 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:47.082340 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:47.096544 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:47.097008 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:47.112977 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.126593 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.129412 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.149791 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:47.212666 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:47.225580 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:47.240799 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.256992 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:47.260020 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:47.279043 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:47.307665 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.481065 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:47.504221 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:47.510715 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:47.515095 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.543431 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.559558 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:47.611351 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:47.692421 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:47.722839 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:47.726298 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:03:47.756897 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.760733 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.762856 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:47.771081 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:47.786302 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:47.788617 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:48.006485 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:48.006675 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:48.008131 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:48.013782 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:48.021759 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:48.024978 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:48.102447 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:48.190269 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:48.230235 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:48.267513 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:48.275729 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:48.284358 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:48.301687 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:48.330868 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:48.355923 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:48.396492 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:48.411584 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:48.483063 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:48.526585 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:48.537485 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:48.557796 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:48.561985 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:48.578995 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:48.581983 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:48.590874 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:48.603703 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:48.614794 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:48.619535 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:48.624397 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:48.651999 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:48.696480 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:48.707024 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:48.822618 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:48.828218 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:48.837917 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:48.859634 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:48.859992 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:48.878295 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:48.883064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.039026 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:49.043848 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:49.050372 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:49.056244 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:49.057257 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:49.065222 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:49.068608 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.068682 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:49.130430 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:49.175885 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:49.215421 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:03:49.241235 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:49.261876 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:03:49.270283 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:49.340263 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:49.345407 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:49.364742 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:49.400907 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.403211 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.412698 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:49.481462 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:03:49.495744 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:49.550552 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:49.582430 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:03:49.642925 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:49.650789 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:49.663085 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:49.670660 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:49.696182 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:49.697871 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.701293 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:49.702261 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:49.730093 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:03:49.738211 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 61293.5us
I20240501 14:03:49.842537 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:49.850761 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:49.856524 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.872310 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.879560 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:49.880434 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:49.982275 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:50.100570 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:50.121588 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.128460 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:50.146401 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.149042 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:50.174141 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:50.177901 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:03:50.272063 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:50.351333 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.354913 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:50.370431 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:50.388245 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:50.388814 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.390851 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.439941 10844 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:03:50.440167 10844 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:50.440862 10844 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:03:50.448143 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:50.454694 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:50.457798 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:50.459678 10844 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:03:50.484079 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:50.497594 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:50.529471 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:03:50.604749 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.608083 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.611634 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:50.623042 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:50.624092 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:50.624442 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:50.633924 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:03:50.679548 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:50.891325 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:50.894193 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:50.904574 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:50.907332 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:50.937808 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:50.956010 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:50.957732 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:50.962474 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:51.022451 7956 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:03:51.022750 7956 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:03:51.023510 7956 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:03:51.024034 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:51.024572 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:03:51.025398 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:03:51.025557 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:03:51.025627 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:03:51.027761 7956 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:51.028352 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:51.045653 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:51.091472 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:51.092209 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:51.127976 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:51.136134 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:51.136420 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:51.167564 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:51.171355 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:51.174410 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:51.196167 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:51.266849 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:51.307333 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:51.358616 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:51.372274 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:51.432399 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:51.436683 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:51.465761 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:51.478659 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:51.522243 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:51.522464 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:51.563668 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:51.600445 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:51.683600 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:51.684717 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:51.698130 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:51.727303 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:51.740110 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:51.763705 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:51.769279 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:51.776808 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:51.806874 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:51.850780 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:51.885913 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:51.952191 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:51.998751 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:52.006870 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.010668 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.019447 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:52.029464 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.039232 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:52.050705 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:52.223179 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:52.290539 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:52.348117 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:52.369840 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:52.389088 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:52.409503 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.409801 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.423381 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:52.494729 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.495177 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:52.520928 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.522325 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.564949 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:52.592180 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:52.668452 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:03:52.752045 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:03:52.785281 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:52.785975 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:52.787329 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.800125 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:52.813769 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:52.814071 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:52.836519 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:03:52.849725 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:53.027629 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:53.034749 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:53.039893 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:53.060117 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:53.062496 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:53.080957 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:03:53.083912 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:53.270656 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:53.281852 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:53.285548 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:53.336161 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:53.340026 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:53.355325 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:53.368682 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:53.401661 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:53.457134 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:53.538682 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:53.546262 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:03:53.574072 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:03:53.575604 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:53.597188 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:53.612380 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:53.626618 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:03:53.627457 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:53.649814 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:53.649924 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:53.651996 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:03:53.666699 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:53.671871 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:53.693535 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:03:53.703831 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:53.807439 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:53.809526 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:53.813964 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:53.819439 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:53.823212 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:53.829820 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:53.890727 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:03:53.912856 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:53.949293 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:54.091223 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:54.092554 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:54.110487 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:54.134335 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:54.135983 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:54.148391 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:54.156157 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:54.267295 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:03:54.296188 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:54.318945 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:54.321803 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:54.331631 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:54.343734 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:54.368906 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:54.369087 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:54.392063 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:54.499279 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:54.536367 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:03:54.543856 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:03:54.544987 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:03:54.604023 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:03:54.616369 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:54.622138 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:54.623157 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:54.629626 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:54.630678 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:54.644261 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:54.656345 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:03:54.686443 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:03:54.788318 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:54.945236 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:54.948124 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:54.969933 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:03:54.971223 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:55.001822 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:55.022049 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.050380 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.071923 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.072360 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:55.078151 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.138875 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:55.153131 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:55.180121 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:55.226112 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:03:55.233170 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:55.389526 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:03:55.400712 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:55.442768 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:55.446944 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:03:55.452440 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:55.475046 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.475716 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.482126 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.504876 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:55.603855 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:03:55.723079 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:55.724771 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:55.748628 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.772599 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:55.780226 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:55.781494 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:55.799333 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:55.928169 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:55.947207 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:03:55.966845 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:55.995146 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:56.006760 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:56.010928 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:03:56.011842 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:56.018993 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.019589 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.034058 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:56.051117 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:56.077492 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:03:56.087469 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:03:56.113343 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:03:56.156759 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:56.245644 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:03:56.259338 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:56.286358 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:56.319433 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:56.327759 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:56.350383 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:56.360559 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.362241 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:56.379016 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:56.391294 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:56.396147 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:03:56.410048 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:56.482074 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.482498 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:56.484184 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.497458 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.509851 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.535379 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:56.647289 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:56.741600 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:56.744016 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:56.745294 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.781517 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.792593 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:56.798308 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:56.802161 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:56.902271 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:56.960444 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:03:56.995946 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:57.002549 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:57.006120 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:57.016788 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:57.029217 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:57.041514 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:57.176985 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:57.223826 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:57.224651 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:03:57.243893 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:57.248943 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:57.262902 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:57.271811 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:57.281262 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:57.282567 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:57.283633 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:57.333699 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:03:57.621695 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:57.627277 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:57.633555 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:03:57.639899 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:57.651414 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:57.672338 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:57.688035 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:57.718278 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:57.725230 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:57.752466 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:57.763593 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:03:57.794005 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 145987us
I20240501 14:03:57.795809 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:57.826879 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:57.854943 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:57.855369 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:57.858780 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:03:57.928299 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 44389.6us
W20240501 14:03:58.007086 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:03:58.093310 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:58.101012 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:58.103263 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:58.118873 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:58.119351 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:58.126308 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:58.270764 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:58.337615 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:58.370893 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:58.389397 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:03:58.394076 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:58.407143 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:58.427762 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:58.447872 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:58.457723 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:58.553406 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:03:58.622474 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:58.626114 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:58.642423 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:58.650959 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:58.665235 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:03:58.671396 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:58.676158 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:58.683017 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:58.707016 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:58.707095 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:58.710927 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:03:58.726053 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:03:58.789645 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:58.848680 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:58.860157 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:58.873821 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:58.978943 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:58.983016 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:03:58.995409 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:03:59.008168 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.020236 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.051429 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:59.056221 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.177776 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.191483 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.198328 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:59.206470 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:59.207762 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:59.239017 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:59.280648 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:03:59.290298 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:59.320729 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:03:59.440802 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:59.446914 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:03:59.463430 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:59.464700 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.503886 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:59.520056 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:03:59.542591 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:03:59.558112 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:03:59.561240 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:59.608443 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:03:59.619385 12124 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:03:59.619629 12124 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:03:59.620302 12124 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:03:59.623303 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:03:59.629467 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:59.629755 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:03:59.630141 12124 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:03:59.680929 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:59.684558 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:59.705305 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:59.712149 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.714609 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:59.721282 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:03:59.723665 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:03:59.727711 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 14:03:59.729446 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:03:59.786644 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:03:59.843876 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:03:59.952248 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:03:59.963337 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:03:59.978029 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.979082 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:03:59.981189 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:03:59.983186 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:00.080130 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:04:00.080560 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:04:00.159284 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:04:00.191567 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:04:00.245899 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:00.250429 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:00.276221 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:00.280828 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:00.283547 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:00.318332 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:00.325870 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:00.329535 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:00.347059 12128 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:04:00.347270 12128 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:04:00.359159 12128 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:04:00.359875 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:04:00.361917 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:04:00.362432 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:00.362545 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:04:00.362942 12128 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:04:00.418392 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:00.429149 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:00.445891 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:00.450156 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:00.461732 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:00.469715 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:00.539876 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:00.620023 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:00.677868 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:00.686753 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:00.694366 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:00.713624 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:00.727931 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:00.752424 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:00.813110 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:04:00.872159 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:00.930619 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:00.953022 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:04:00.963546 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:00.976630 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:00.976889 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:00.985656 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:04:00.995663 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:00.999835 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:01.021322 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:01.103324 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:01.108760 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:04:01.147915 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:04:01.149468 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:01.240710 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:01.253249 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:04:01.255681 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:01.279109 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:01.296634 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:01.302384 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:01.305732 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:01.327528 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:01.335758 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:01.352854 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:04:01.505666 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:04:01.535096 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:01.548547 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:01.565495 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:01.598153 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:01.599836 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:01.604777 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:01.643896 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:01.735137 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:01.735919 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:01.759829 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:01.779878 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:01.780778 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:01.798676 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:01.824592 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:01.891772 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:04:01.905098 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:01.938750 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:04:01.967275 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:04:02.030900 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:02.046911 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.055486 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:02.056739 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.057133 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.090561 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:02.203231 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:02.255074 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:02.259410 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:02.271034 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.279048 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:02.294569 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.322747 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:02.419565 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:02.512483 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:02.523970 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:02.567287 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.579701 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.586998 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:02.593220 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:02.597654 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:02.620254 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:02.636363 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:04:02.773833 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:04:02.791829 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:04:02.853412 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.857740 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:02.860216 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:02.889870 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.893493 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:02.916456 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:02.951436 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:02.958258 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:02.988011 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:02.990504 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:02.993232 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:03.024909 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:03.027150 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.030972 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:03.049186 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.286140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:03.287876 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:03.291373 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:03.311575 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.313146 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.316828 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:03.378549 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:03.395138 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:03.404930 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:03.494282 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:03.498314 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:03.525792 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:03.579677 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.580659 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:03.583312 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:03.600366 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:03.601418 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.604182 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:03.642359 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:03.714712 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:03.816150 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:03.838443 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:03.842095 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.844648 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:03.848767 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:03.854424 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:03.863903 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:03.866346 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:03.929914 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:03.952556 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:04.030817 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:04.094110 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:04.109047 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:04.136067 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.158615 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:04.202942 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.220439 28593 maintenance_manager.cc:419] P 1bdce3de4d694d6c98f909a5b3c450ce: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033387
W20240501 14:04:04.227162 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:04:04.236403 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:04.252341 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:04.258675 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:04:04.304543 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.305507 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:04.306895 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:04.360853 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.380447 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.435585 28592 maintenance_manager.cc:643] P 1bdce3de4d694d6c98f909a5b3c450ce: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.214s user 0.047s sys 0.004s Metrics: {"bytes_written":20275,"cfile_init":1,"dirs.queue_time_us":13631,"dirs.run_cpu_time_us":335,"dirs.run_wall_time_us":11039,"drs_written":1,"lbm_read_time_us":205,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1423,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":20238,"threads_started":2,"wal-append.queue_time_us":6149}
I20240501 14:04:04.438062 28593 maintenance_manager.cc:419] P 1bdce3de4d694d6c98f909a5b3c450ce: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7636 bytes on disk
I20240501 14:04:04.442044 28592 maintenance_manager.cc:643] P 1bdce3de4d694d6c98f909a5b3c450ce: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 14:04:04.458500 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:04.460269 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:04.569509 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:04.579371 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:04.624951 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:04:04.635254 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:04:04.693327 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:04.706532 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:04.707449 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:04.710110 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.722429 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.744478 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:04.785372 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:04.872047 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:04:04.946902 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:04.973006 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:04.974455 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.976140 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:04.995560 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:05.001240 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:05.030769 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:04:05.043593 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:05.077890 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:05.082986 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:05.210031 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:05.214411 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:05.243829 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:05.248503 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:05.250982 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:04:05.252911 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:05.259675 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:05.272969 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:05.290244 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:05.348544 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:04:05.449723 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:04:05.595695 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:05.626616 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:05.633720 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:05.634493 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:05.674324 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:05.686205 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:05.723323 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:05.751385 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:05.779649 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:05.791620 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:05.814698 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:04:05.820091 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:05.856896 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:05.874889 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:05.878417 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:05.891988 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:05.902611 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:05.929375 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:05.934207 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:06.010167 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:06.101964 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:06.126892 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:06.129597 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:06.134150 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:06.157431 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:06.164232 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:06.175892 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:06.258167 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:06.367466 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:06.381421 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:06.383952 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:06.388207 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:06.389983 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:06.420451 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:06.421419 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:06.421799 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:06.463364 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:06.465787 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:06.565992 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:06.566598 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:04:06.567902 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:06.584650 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:06.650786 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:06.684716 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:06.689126 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:06.690701 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:06.715338 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:06.724005 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:06.736481 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:06.785622 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:06.903057 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:04:06.971168 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:06.992147 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:07.003733 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.014282 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:07.015282 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:07.037369 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:07.046773 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:07.121668 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:04:07.141606 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:07.144253 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:07.162359 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.177513 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:07.183467 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.196921 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.445000 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:07.447232 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:07.456797 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:07.476138 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:07.487921 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.488942 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.496532 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:07.522048 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:07.546277 30729 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 82644.2us
W20240501 14:04:07.555191 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:07.711652 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:07.740500 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:07.745018 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:07.746038 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.750500 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:07.750519 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:07.758220 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:07.759299 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:07.778791 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:07.845628 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:08.021492 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.026361 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.027323 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.039392 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:08.053061 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:08.062204 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:08.182713 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:04:08.201939 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:08.263854 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:08.327184 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:08.328367 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.330490 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:08.340775 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:08.342406 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:08.364252 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:08.404829 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:08.407121 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:08.415633 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:08.444661 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:08.468027 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:08.513433 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:08.547557 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:08.557009 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:08.557880 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:08.566246 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:08.578336 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:08.581806 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.588140 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.595908 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.603658 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:08.758067 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:08.855376 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:08.865996 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.883567 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.891439 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:08.907346 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:08.921916 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:08.922822 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:08.955192 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:08.994834 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:09.053238 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:09.058688 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:09.060684 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.077064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.084985 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:09.085690 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:09.132683 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:09.151789 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:09.176402 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:09.272104 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:09.355088 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:09.361825 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.373641 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.375399 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.392463 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:09.424221 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:09.447656 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:09.640900 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:04:09.662283 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:04:09.671718 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:09.675792 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:09.696255 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.699609 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.702556 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:09.783049 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:09.783459 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:09.799465 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:09.824728 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:09.830652 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:09.847071 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:09.867688 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.910025 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:09.920503 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:09.970909 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:09.983536 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:04:10.126765 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:10.133098 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:10.145085 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:10.160408 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:10.177364 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:10.186247 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:04:10.186681 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:10.199601 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:10.207887 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:04:10.208673 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:10.252600 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:04:10.253991 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:10.305238 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:10.473191 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:10.485615 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:10.491493 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:10.503414 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:10.516872 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:10.540997 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:04:10.547900 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:10.703475 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:10.712466 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:10.714959 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:10.722896 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:10.745515 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:10.772017 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:10.792971 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:10.798449 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:10.834343 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:10.834908 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:10.838474 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:10.884012 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:10.884166 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:10.905720 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:10.916611 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:11.004827 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:11.032574 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:11.034046 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:11.038394 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:11.052575 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:11.075644 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:11.094039 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:11.109053 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:11.123723 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:11.137773 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:11.157737 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:11.195037 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:11.203428 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:11.245625 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:11.283095 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:11.401774 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:11.458665 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:11.506995 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:11.509234 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:11.536154 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:11.547454 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:11.553637 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:11.555783 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:11.565366 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:11.580592 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:11.581805 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:11.749068 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:11.770983 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:11.786584 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:11.787170 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:11.806047 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:11.806617 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:11.820082 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:11.821473 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:04:11.824337 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:12.024973 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:12.030301 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:12.033142 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:12.035423 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:12.038239 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:12.045778 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:12.071815 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:12.103641 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:12.185436 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:12.312888 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:12.360855 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:12.426208 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:12.467607 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:12.476912 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:12.503391 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:12.505060 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:12.506886 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:12.513036 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:12.515412 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:12.550695 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:12.553272 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:12.576833 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:12.604252 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:12.631778 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:12.658977 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(86bfedcdabe44e7ab135674163935786): perf score=0.033336
W20240501 14:04:12.683318 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:12.701241 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:04:12.714583 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:12.766183 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(86bfedcdabe44e7ab135674163935786) complete. Timing: real 0.105s user 0.060s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":20193,"dirs.queue_time_us":5204,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":2598,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":672,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26841,"threads_started":3,"wal-append.queue_time_us":391}
I20240501 14:04:12.768568 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(86bfedcdabe44e7ab135674163935786): 4834 bytes on disk
I20240501 14:04:12.769588 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(86bfedcdabe44e7ab135674163935786) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240501 14:04:12.785657 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:04:12.839951 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(bb712628c7764a138f5425d14b724d9e): perf score=0.033393
I20240501 14:04:12.922215 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:12.963852 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:12.982164 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:13.032649 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:13.033854 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:13.053452 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:13.063773 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:13.085507 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(bb712628c7764a138f5425d14b724d9e) complete. Timing: real 0.245s user 0.080s sys 0.034s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":7655,"dirs.queue_time_us":26385,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":8502,"drs_written":1,"lbm_read_time_us":168,"lbm_reads_lt_1ms":4,"lbm_write_time_us":990,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":117815,"threads_started":3,"wal-append.queue_time_us":64855}
I20240501 14:04:13.095396 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(bb712628c7764a138f5425d14b724d9e): 4581 bytes on disk
I20240501 14:04:13.101101 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(bb712628c7764a138f5425d14b724d9e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
W20240501 14:04:13.178944 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 64892.6us
W20240501 14:04:13.183122 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:13.295017 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:13.334951 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:13.399065 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:13.410509 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:13.419343 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:13.435326 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:13.437121 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:13.437808 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:13.456594 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:13.464170 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:13.481097 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:13.500702 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:13.526940 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:13.550550 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:13.575822 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:13.580066 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:13.674566 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:13.686049 14323 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:04:13.686264 14323 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:04:13.687031 14323 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:04:13.688061 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:04:13.688860 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:04:13.689513 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:13.689657 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:04:13.691215 14323 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:04:13.729733 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:13.803035 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:13.810034 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:13.817386 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:13.826824 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:13.832139 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:13.835675 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:13.852277 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:13.878801 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:14.130501 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:14.130960 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:14.144059 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:14.145296 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:14.209537 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:14.226893 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:14.256525 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:14.281888 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:14.329236 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:14.351028 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:14.360152 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:14.384212 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:14.400331 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:14.406332 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:14.408160 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:14.426573 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:14.448660 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:14.614387 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(fb04ec77da194832aa63d69aba307de3): perf score=0.033337
I20240501 14:04:14.674886 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:14.677445 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:14.701535 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:14.716624 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:04:14.744210 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:04:14.754982 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:14.778731 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(f516f01e242d4b8789130201f748f3ea): perf score=0.033379
I20240501 14:04:14.820051 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:14.824611 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:14.833914 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(fb04ec77da194832aa63d69aba307de3): perf score=0.033390
W20240501 14:04:14.867946 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:14.885275 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:04:14.891388 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(fb04ec77da194832aa63d69aba307de3) complete. Timing: real 0.276s user 0.102s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":1092,"dirs.queue_time_us":15670,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":5958,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4479,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86590,"threads_started":3,"wal-append.queue_time_us":31495}
I20240501 14:04:14.893724 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(fb04ec77da194832aa63d69aba307de3): 4834 bytes on disk
I20240501 14:04:14.895639 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(fb04ec77da194832aa63d69aba307de3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240501 14:04:14.919529 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(f516f01e242d4b8789130201f748f3ea) complete. Timing: real 0.140s user 0.038s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3703,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":3087,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11765,"threads_started":2,"wal-append.queue_time_us":7749}
I20240501 14:04:14.921418 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(f516f01e242d4b8789130201f748f3ea): 4581 bytes on disk
I20240501 14:04:14.923506 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(f516f01e242d4b8789130201f748f3ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20240501 14:04:14.942493 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:14.983662 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(fb04ec77da194832aa63d69aba307de3) complete. Timing: real 0.149s user 0.064s sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":17817,"dirs.queue_time_us":6792,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":4006,"drs_written":1,"lbm_read_time_us":172,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1002,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":276864,"thread_start_us":33054,"threads_started":3,"wal-append.queue_time_us":7047}
I20240501 14:04:14.985915 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(fb04ec77da194832aa63d69aba307de3): 4834 bytes on disk
I20240501 14:04:14.986986 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(fb04ec77da194832aa63d69aba307de3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
W20240501 14:04:15.042924 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:15.080513 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:15.102543 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:04:15.118523 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:15.145206 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:15.152115 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:15.170287 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:15.192520 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:15.218997 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:04:15.227154 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:15.308396 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:15.349869 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:04:15.433593 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:15.576901 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:04:15.578083 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:15.637388 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:15.702455 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:15.735977 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:15.802685 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:15.865063 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(6feda9b53e484e7ea40a5890b7d51ac4): perf score=0.033345
I20240501 14:04:15.870286 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:15.899837 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:15.900427 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(e0bf08cbb6d347d783b1fa7762c802df): perf score=0.033356
W20240501 14:04:15.919246 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 53011.5us
I20240501 14:04:15.923736 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:15.925858 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:15.944644 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:15.959635 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:15.992030 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(014635e01a22420b8bc21e879449749b): perf score=0.033380
W20240501 14:04:16.017427 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:16.018445 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:16.018802 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:16.028838 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(6feda9b53e484e7ea40a5890b7d51ac4) complete. Timing: real 0.163s user 0.077s sys 0.045s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":64741,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":1837,"drs_written":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1042,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":75505,"threads_started":2,"wal-append.queue_time_us":10701}
I20240501 14:04:16.038070 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4): 4921 bytes on disk
I20240501 14:04:16.038770 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 14:04:16.040477 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:16.045644 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(e0bf08cbb6d347d783b1fa7762c802df) complete. Timing: real 0.145s user 0.042s sys 0.039s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34481,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":4248,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1093,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":43005,"threads_started":2,"wal-append.queue_time_us":6479}
I20240501 14:04:16.046955 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(e0bf08cbb6d347d783b1fa7762c802df): 4581 bytes on disk
I20240501 14:04:16.051301 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(e0bf08cbb6d347d783b1fa7762c802df) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4}
W20240501 14:04:16.062322 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:16.075093 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:16.080088 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:16.087512 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(014635e01a22420b8bc21e879449749b) complete. Timing: real 0.095s user 0.051s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":20350,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":1482,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":855,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29228,"threads_started":2,"wal-append.queue_time_us":8950}
I20240501 14:04:16.089180 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(014635e01a22420b8bc21e879449749b): 4834 bytes on disk
I20240501 14:04:16.089910 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(014635e01a22420b8bc21e879449749b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240501 14:04:16.131774 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:16.188760 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:16.198189 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:16.206907 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:16.212247 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:16.255673 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:16.288862 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:16.291275 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:16.356812 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:16.409690 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:16.486267 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:16.493188 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:16.500659 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:16.503142 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:16.504716 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:16.515522 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:16.532729 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:16.548688 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:16.693501 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:16.766117 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:16.779142 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:16.802995 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:16.814790 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:16.824385 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:16.845947 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f): perf score=0.033338
I20240501 14:04:16.871938 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:16.907061 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:16.934589 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(63b7f2a8d8a84faf86c4de16b3cc6663): perf score=0.033360
I20240501 14:04:16.982024 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.136s user 0.058s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6393,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":13191,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2161,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":60720,"threads_started":2,"wal-append.queue_time_us":54433}
I20240501 14:04:16.984244 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(e9bb3eaf3f0e40f8902b9426a845d9b2): perf score=0.033371
I20240501 14:04:17.020792 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(63b7f2a8d8a84faf86c4de16b3cc6663) complete. Timing: real 0.086s user 0.043s sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2197,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1522,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":27292,"threads_started":2,"wal-append.queue_time_us":25005}
I20240501 14:04:17.022594 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(63b7f2a8d8a84faf86c4de16b3cc6663): 4834 bytes on disk
I20240501 14:04:17.023490 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(63b7f2a8d8a84faf86c4de16b3cc6663) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":200,"lbm_reads_lt_1ms":4}
I20240501 14:04:17.044066 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f): perf score=0.033388
I20240501 14:04:17.053599 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(e9bb3eaf3f0e40f8902b9426a845d9b2) complete. Timing: real 0.068s user 0.027s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1686,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1073,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6757,"threads_started":1,"wal-append.queue_time_us":6625}
I20240501 14:04:17.055344 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f): 4921 bytes on disk
I20240501 14:04:17.056031 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 14:04:17.057328 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(e9bb3eaf3f0e40f8902b9426a845d9b2): 4581 bytes on disk
I20240501 14:04:17.057912 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(e9bb3eaf3f0e40f8902b9426a845d9b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:04:17.058326 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(e9bb3eaf3f0e40f8902b9426a845d9b2): perf score=0.033399
W20240501 14:04:17.080387 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:04:17.089509 15044 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:04:17.089743 15044 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:04:17.090556 15044 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:04:17.092645 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:17.096964 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:17.097167 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:04:17.097438 15044 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:04:17.100574 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:17.109400 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:17.136039 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:17.154405 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:17.165544 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.121s user 0.056s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7421,"dirs.run_cpu_time_us":294,"dirs.run_wall_time_us":38588,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1002,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16211,"threads_started":2,"wal-append.queue_time_us":8482}
I20240501 14:04:17.167421 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f): 4921 bytes on disk
I20240501 14:04:17.168432 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4}
I20240501 14:04:17.184914 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(e9bb3eaf3f0e40f8902b9426a845d9b2) complete. Timing: real 0.126s user 0.039s sys 0.044s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":17432,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":12370,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8129,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":61423,"threads_started":2,"wal-append.queue_time_us":43053}
I20240501 14:04:17.189693 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(e9bb3eaf3f0e40f8902b9426a845d9b2): 4581 bytes on disk
I20240501 14:04:17.192365 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(e9bb3eaf3f0e40f8902b9426a845d9b2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240501 14:04:17.242535 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:17.271692 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:17.288769 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:17.288872 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:17.328694 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:17.360103 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:17.455303 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:17.456808 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:17.467572 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:17.469472 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:17.491648 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:17.499537 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:17.503521 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:17.750113 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:04:17.771951 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:04:17.782884 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:04:17.860087 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:17.864228 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:17.868674 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:17.882936 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:17.906991 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:17.912684 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:17.931253 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:17.940385 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:17.942664 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:17.943771 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:17.985913 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:18.010608 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:18.044936 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:18.087801 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:04:18.121690 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:18.173766 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(de161b10c84f4bcc85f8cf7b199f99db): perf score=0.033333
W20240501 14:04:18.195986 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:18.197693 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(39f1ab3a94db41b08ac90bba52418a93): perf score=0.033361
W20240501 14:04:18.273854 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:18.299156 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:18.302392 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:18.306823 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(de161b10c84f4bcc85f8cf7b199f99db) complete. Timing: real 0.133s user 0.053s sys 0.035s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":74304,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":9602,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":951,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":75910,"threads_started":2,"wal-append.queue_time_us":2495}
I20240501 14:04:18.308815 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(de161b10c84f4bcc85f8cf7b199f99db): 4581 bytes on disk
I20240501 14:04:18.310395 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(de161b10c84f4bcc85f8cf7b199f99db) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 14:04:18.316412 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(39f1ab3a94db41b08ac90bba52418a93): perf score=0.033370
W20240501 14:04:18.340304 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:18.359719 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:18.365286 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:18.375761 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:18.388967 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:18.397838 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:18.399427 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:18.426244 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(39f1ab3a94db41b08ac90bba52418a93) complete. Timing: real 0.228s user 0.049s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":372,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":2513,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12681,"threads_started":2,"wal-append.queue_time_us":12315}
I20240501 14:04:18.455919 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(12510f71b6a742ab8cc359243de8119b): perf score=0.033422
I20240501 14:04:18.483429 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(39f1ab3a94db41b08ac90bba52418a93) complete. Timing: real 0.167s user 0.063s sys 0.054s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":23698,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":6791,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":73653,"threads_started":2,"wal-append.queue_time_us":50193}
I20240501 14:04:18.485153 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93): 4834 bytes on disk
I20240501 14:04:18.491225 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240501 14:04:18.503782 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:04:18.535444 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:18.543889 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:18.544987 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(12510f71b6a742ab8cc359243de8119b) complete. Timing: real 0.088s user 0.051s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":2670,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":984,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":221,"threads_started":1,"wal-append.queue_time_us":344}
I20240501 14:04:18.546027 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(12510f71b6a742ab8cc359243de8119b): 4921 bytes on disk
I20240501 14:04:18.546690 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(12510f71b6a742ab8cc359243de8119b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:04:18.547542 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93): 4834 bytes on disk
I20240501 14:04:18.548044 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240501 14:04:18.627130 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:18.638185 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:18.644639 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:18.644912 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:18.659516 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:18.667533 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:18.672497 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:18.672518 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:18.680824 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:18.717144 14915 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:04:18.717334 14915 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:04:18.718125 14915 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:04:18.720014 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:04:18.721032 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:04:18.721634 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:18.722893 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:18.723007 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:04:18.723477 14915 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:04:18.757748 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:18.815639 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:18.894732 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:18.901582 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:18.903512 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:18.921851 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:18.922482 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:18.944221 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:19.019547 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:19.138005 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:19.186568 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:19.210223 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:19.236454 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:19.239013 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.243400 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:19.264123 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.290036 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:19.317299 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:19.345515 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:19.352506 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.354434 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.367748 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.393232 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.427182 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:19.500137 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(6aaf52ee71b44511b7b8909711ab0a42): perf score=0.033366
W20240501 14:04:19.514572 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:19.537120 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(f4402de0b48041c383568feb9365147e): perf score=0.033377
I20240501 14:04:19.566421 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(7c0bc741ed064dd7ad8ec70219496c6f): perf score=0.033381
I20240501 14:04:19.611797 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(6aaf52ee71b44511b7b8909711ab0a42) complete. Timing: real 0.111s user 0.043s sys 0.036s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14283,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":28579,"threads_started":2,"wal-append.queue_time_us":14539}
I20240501 14:04:19.613664 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(6aaf52ee71b44511b7b8909711ab0a42): 4834 bytes on disk
I20240501 14:04:19.615731 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(6aaf52ee71b44511b7b8909711ab0a42) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 14:04:19.623428 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(f4402de0b48041c383568feb9365147e) complete. Timing: real 0.086s user 0.058s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7568,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":5647,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":998,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11883,"threads_started":2,"wal-append.queue_time_us":4539}
I20240501 14:04:19.624679 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(f4402de0b48041c383568feb9365147e): 4581 bytes on disk
I20240501 14:04:19.625330 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(f4402de0b48041c383568feb9365147e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 14:04:19.650996 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(7c0bc741ed064dd7ad8ec70219496c6f) complete. Timing: real 0.084s user 0.062s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":351,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1397,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":490,"threads_started":2,"wal-append.queue_time_us":363}
I20240501 14:04:19.652187 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(7c0bc741ed064dd7ad8ec70219496c6f): 4921 bytes on disk
I20240501 14:04:19.652745 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(7c0bc741ed064dd7ad8ec70219496c6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 14:04:19.706648 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:19.721571 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.724565 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.737253 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:19.760429 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:19.762954 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.776165 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:19.785677 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:04:19.887104 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:04:19.940299 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:04:19.953552 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:04:19.991266 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:19.993202 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:20.006135 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:20.010905 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:20.015451 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:20.019713 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:20.084542 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:20.113359 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:04:20.161576 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:20.238613 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:20.240857 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 14:04:20.248639 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:20.254068 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:20.265153 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:20.284602 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:20.285045 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:20.337270 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:20.354465 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:04:20.399747 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:20.483130 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:20.556488 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(6b6a54ac32694640bffdf1c7f3b035cc): perf score=0.033348
I20240501 14:04:20.584594 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:20.597216 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:20.616254 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:20.621287 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(763b2e84afa64924ad5eeeeb3265a1d8): perf score=0.033373
W20240501 14:04:20.628229 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:20.628582 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:20.629756 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(6b6a54ac32694640bffdf1c7f3b035cc): perf score=0.033378
I20240501 14:04:20.658150 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(763b2e84afa64924ad5eeeeb3265a1d8): perf score=0.033379
I20240501 14:04:20.669075 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:20.695048 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 69538.6us
W20240501 14:04:20.711360 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:20.723925 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:20.726214 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:20.748482 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:20.763607 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 58437.6us
I20240501 14:04:20.792896 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:20.800559 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:20.805335 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(6b6a54ac32694640bffdf1c7f3b035cc) complete. Timing: real 0.175s user 0.041s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9206,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":4267,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":95373,"threads_started":2,"wal-append.queue_time_us":85644}
I20240501 14:04:20.806581 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(6b6a54ac32694640bffdf1c7f3b035cc): 4579 bytes on disk
W20240501 14:04:20.807432 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:04:20.809769 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:20.820250 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(6b6a54ac32694640bffdf1c7f3b035cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
W20240501 14:04:20.834172 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:20.836020 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:20.862612 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:20.869041 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:20.872995 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:20.875111 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(763b2e84afa64924ad5eeeeb3265a1d8) complete. Timing: real 0.254s user 0.052s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42897,"dirs.run_cpu_time_us":303,"dirs.run_wall_time_us":11729,"drs_written":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1034,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":74312,"threads_started":2,"wal-append.queue_time_us":26461}
I20240501 14:04:20.876879 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(135da52054584a27959efdd9126cfcef): perf score=0.033439
W20240501 14:04:20.879362 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:20.894994 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:20.960286 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(135da52054584a27959efdd9126cfcef) complete. Timing: real 0.075s user 0.036s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":109,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":1565,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":889,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":241,"threads_started":1,"wal-append.queue_time_us":348}
I20240501 14:04:20.961855 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef): 4921 bytes on disk
I20240501 14:04:20.962641 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240501 14:04:20.963974 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(763b2e84afa64924ad5eeeeb3265a1d8): 4834 bytes on disk
I20240501 14:04:20.964538 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(763b2e84afa64924ad5eeeeb3265a1d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 14:04:20.968624 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(6b6a54ac32694640bffdf1c7f3b035cc) complete. Timing: real 0.412s user 0.076s sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":14097,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":8429,"drs_written":1,"lbm_read_time_us":157,"lbm_reads_lt_1ms":4,"lbm_write_time_us":41084,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":58594,"threads_started":2,"wal-append.queue_time_us":44461}
I20240501 14:04:20.970057 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(6b6a54ac32694640bffdf1c7f3b035cc): 4579 bytes on disk
I20240501 14:04:20.970770 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(6b6a54ac32694640bffdf1c7f3b035cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
W20240501 14:04:20.992822 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:20.998909 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(763b2e84afa64924ad5eeeeb3265a1d8) complete. Timing: real 0.340s user 0.033s sys 0.045s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":60271,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":5123,"drs_written":1,"lbm_read_time_us":66648,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":16764,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":72550,"threads_started":2,"wal-append.queue_time_us":9921}
I20240501 14:04:21.000252 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(135da52054584a27959efdd9126cfcef): perf score=0.033471
I20240501 14:04:21.038573 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(135da52054584a27959efdd9126cfcef) complete. Timing: real 0.038s user 0.023s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":658,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":316,"threads_started":1,"wal-append.queue_time_us":429}
I20240501 14:04:21.044044 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef): 4921 bytes on disk
I20240501 14:04:21.045090 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 14:04:21.045962 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(763b2e84afa64924ad5eeeeb3265a1d8): 4834 bytes on disk
I20240501 14:04:21.046516 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(763b2e84afa64924ad5eeeeb3265a1d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240501 14:04:21.074204 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:21.084156 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:21.140337 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:21.160878 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:21.164752 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:21.166843 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:21.174129 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:21.194098 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:21.201630 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:21.203003 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:21.242918 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:21.382596 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:21.447656 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:21.451022 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:21.453042 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:21.471589 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:21.487640 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:21.489302 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:21.506738 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:21.549531 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(e821b93aa9b84ff3a968860a5b33b918): perf score=0.033343
I20240501 14:04:21.574749 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(4d8c435795a04b1b90b6e2ff914676c8): perf score=0.033346
W20240501 14:04:21.598162 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:21.643811 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(e821b93aa9b84ff3a968860a5b33b918) complete. Timing: real 0.094s user 0.063s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6584,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":6611,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1127,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6958,"threads_started":2,"wal-append.queue_time_us":339}
I20240501 14:04:21.645267 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(e821b93aa9b84ff3a968860a5b33b918): 4921 bytes on disk
I20240501 14:04:21.646288 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(e821b93aa9b84ff3a968860a5b33b918) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 14:04:21.653120 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(4d8c435795a04b1b90b6e2ff914676c8) complete. Timing: real 0.078s user 0.041s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":17785,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1573,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":893,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17925,"threads_started":2,"wal-append.queue_time_us":382}
W20240501 14:04:21.653401 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:21.654186 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(4d8c435795a04b1b90b6e2ff914676c8): 4834 bytes on disk
I20240501 14:04:21.654922 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(4d8c435795a04b1b90b6e2ff914676c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4}
I20240501 14:04:21.718688 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(e821b93aa9b84ff3a968860a5b33b918): perf score=0.033391
I20240501 14:04:21.767904 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:21.785096 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:21.789914 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:21.823635 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:21.828598 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(e821b93aa9b84ff3a968860a5b33b918) complete. Timing: real 0.109s user 0.079s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9006,"dirs.run_cpu_time_us":318,"dirs.run_wall_time_us":8826,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":915,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16062,"threads_started":2,"wal-append.queue_time_us":6731}
I20240501 14:04:21.830271 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(e821b93aa9b84ff3a968860a5b33b918): 4921 bytes on disk
I20240501 14:04:21.831358 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(e821b93aa9b84ff3a968860a5b33b918) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240501 14:04:21.871265 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 78578.6us
I20240501 14:04:21.871394 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:21.874164 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:21.875684 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:21.962432 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:22.125394 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:22.133616 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.140367 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:22.153039 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:22.157193 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.172210 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:22.276201 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:04:22.310902 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:22.318439 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:22.328245 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:22.343778 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.347558 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:22.386768 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:22.408170 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:22.411551 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:22.490665 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:22.584115 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:22.603430 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:22.614594 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:22.628849 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:22.666980 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.667498 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.677685 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:22.681983 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.728405 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(efe7fed8e5d44e8a80ffbdc27619a74d): perf score=0.033334
I20240501 14:04:22.824491 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(efe7fed8e5d44e8a80ffbdc27619a74d) complete. Timing: real 0.096s user 0.033s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8205,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":10861,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":977,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8570,"threads_started":2,"wal-append.queue_time_us":323}
I20240501 14:04:22.825609 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(efe7fed8e5d44e8a80ffbdc27619a74d): 4834 bytes on disk
I20240501 14:04:22.826114 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(efe7fed8e5d44e8a80ffbdc27619a74d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240501 14:04:22.841012 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:04:22.886165 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:22.899448 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.902386 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(9b7c7b963588436bb778e45da469c7d5): perf score=0.033382
I20240501 14:04:22.906250 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:22.917269 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:22.926097 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.932035 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:22.997505 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(9b7c7b963588436bb778e45da469c7d5) complete. Timing: real 0.095s user 0.045s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":342,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":1423,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":861,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":456,"threads_started":2,"wal-append.queue_time_us":355}
I20240501 14:04:22.998906 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(9b7c7b963588436bb778e45da469c7d5): 4581 bytes on disk
I20240501 14:04:23.000370 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(9b7c7b963588436bb778e45da469c7d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240501 14:04:23.002327 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:23.037034 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:04:23.122555 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:23.138159 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:23.158689 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:23.159287 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:23.167923 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:23.168175 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:23.178779 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:23.191651 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:23.197466 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:23.275197 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:23.320001 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:23.321439 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:23.448063 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:23.452667 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:23.474325 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:23.493263 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:23.526839 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:23.527746 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:23.581600 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:23.595011 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:23.611881 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:23.627085 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:23.640183 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:23.644603 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:23.670673 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:23.688603 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:23.715682 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:23.719249 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:23.730226 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:23.748546 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:23.761040 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:23.765286 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:23.891129 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:23.918165 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(111ba172d7e44054872c8fc8cac8ba88): perf score=0.033336
W20240501 14:04:23.925391 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:23.947551 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:23.980295 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(111ba172d7e44054872c8fc8cac8ba88) complete. Timing: real 0.062s user 0.044s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":2680,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":532,"threads_started":2,"wal-append.queue_time_us":371}
I20240501 14:04:23.981555 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(1fdedd39e12a49659d989f37b3287d87): perf score=0.033349
I20240501 14:04:23.986815 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:23.991115 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:23.993800 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:24.000872 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.029992 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:24.030828 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.067610 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.081852 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(111ba172d7e44054872c8fc8cac8ba88): perf score=0.033386
I20240501 14:04:24.092113 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(9b473e10a8ad4bd58bb44aabd9656e08): perf score=0.033390
I20240501 14:04:24.114949 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(1fdedd39e12a49659d989f37b3287d87) complete. Timing: real 0.133s user 0.041s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1166,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11191,"threads_started":1,"wal-append.queue_time_us":10372}
I20240501 14:04:24.117777 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(111ba172d7e44054872c8fc8cac8ba88): 4922 bytes on disk
I20240501 14:04:24.119354 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(111ba172d7e44054872c8fc8cac8ba88) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":341,"lbm_reads_lt_1ms":4}
I20240501 14:04:24.120702 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(1fdedd39e12a49659d989f37b3287d87): 4581 bytes on disk
I20240501 14:04:24.121330 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(1fdedd39e12a49659d989f37b3287d87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240501 14:04:24.156011 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:24.170110 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(111ba172d7e44054872c8fc8cac8ba88) complete. Timing: real 0.088s user 0.030s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3862,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":7413,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1009,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5657,"threads_started":2,"wal-append.queue_time_us":1552}
I20240501 14:04:24.171631 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(111ba172d7e44054872c8fc8cac8ba88): 4922 bytes on disk
I20240501 14:04:24.172309 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(111ba172d7e44054872c8fc8cac8ba88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 14:04:24.173543 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(9b473e10a8ad4bd58bb44aabd9656e08): perf score=0.033408
I20240501 14:04:24.199896 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(9b473e10a8ad4bd58bb44aabd9656e08) complete. Timing: real 0.107s user 0.049s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9991,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1967,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":893,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13913,"threads_started":2,"wal-append.queue_time_us":4135}
I20240501 14:04:24.201714 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08): 4834 bytes on disk
I20240501 14:04:24.203351 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240501 14:04:24.236152 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(9b473e10a8ad4bd58bb44aabd9656e08) complete. Timing: real 0.062s user 0.032s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":83,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":1381,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7048,"threads_started":1,"wal-append.queue_time_us":6916}
I20240501 14:04:24.237357 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08): 4834 bytes on disk
I20240501 14:04:24.238376 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 14:04:24.246577 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:24.310324 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:24.326334 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.336107 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.345185 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:24.351670 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.369812 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:24.587606 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:24.598343 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.603080 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.616050 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:24.645882 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:24.646538 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:24.730571 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:24.828621 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:04:24.832602 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:24.887272 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:04:24.888321 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:24.945057 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:24.952559 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:24.960628 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.004575 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:25.041837 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:25.056327 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:04:25.060312 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.083216 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:25.094168 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:25.100276 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:25.119541 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:04:25.120776 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:25.177678 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.184616 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.209046 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.249667 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:25.328716 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:25.417798 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:25.431244 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:04:25.467545 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:04:25.468544 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.472798 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:25.475914 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.495815 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:25.516885 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:25.518775 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:25.539232 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:25.666719 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:25.710812 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:25.726929 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:25.727990 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:25.729861 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:25.731954 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:25.745978 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:25.770125 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:25.770980 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:25.778250 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:25.845697 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:04:25.905790 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:25.958117 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:25.962234 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:25.965827 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:25.982484 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:25.994992 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:25.995008 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:26.007107 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:26.038389 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:26.054289 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:04:26.056226 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:04:26.108364 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:26.153894 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:26.203066 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:04:26.225953 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:26.278932 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:26.281910 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:26.286849 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:26.336951 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:26.354677 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:26.361145 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:26.408255 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:26.443217 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:26.445276 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:26.468222 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:26.479959 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:26.503001 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:26.542748 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:26.543661 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:26.580641 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:26.583583 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:26.762933 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:26.812510 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:26.812614 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:26.820713 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:26.852360 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:26.869282 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:26.869978 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:26.899628 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:26.975950 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:27.093070 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:27.094116 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:27.095006 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:27.097391 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:27.103207 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:27.108369 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:27.153574 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:27.312945 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:27.356348 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:27.364109 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:27.384442 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:27.387984 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:27.389529 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:27.416303 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:27.512751 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:27.548310 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:27.738274 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:27.744334 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:27.777035 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:27.813604 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 92586.1us
W20240501 14:04:27.819475 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:27.833981 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:27.836622 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:27.837217 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:27.883595 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:27.903314 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:04:27.912308 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:04:27.912454 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:27.920670 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:27.955487 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:27.960222 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 47920.4us
I20240501 14:04:27.964823 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:28.009378 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:28.022023 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:28.049758 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:28.056255 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:28.064821 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:28.175520 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:04:28.246024 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:28.248198 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:28.268668 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:28.275163 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:28.275761 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:28.304316 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:28.304345 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:28.317808 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:28.318225 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:28.425334 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:28.484095 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:28.507395 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:28.524032 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:28.525797 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:28.528231 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:28.529284 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:28.553174 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:28.570678 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:28.590970 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:28.606006 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:28.616376 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:04:28.715766 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:28.793771 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:28.795158 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:28.803639 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:28.807157 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:28.822516 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:28.826436 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:28.837738 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:28.901296 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:28.917856 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:04:28.956477 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:29.100881 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:29.101713 16218 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:04:29.101919 16218 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:04:29.102751 16218 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:04:29.104269 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:04:29.105268 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:04:29.106541 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:29.106704 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:04:29.107317 16218 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:04:29.121516 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:29.128310 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:29.135722 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:29.143350 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:29.176364 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:29.192036 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:29.195091 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:29.203311 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:29.220821 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:29.300539 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:29.313313 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:29.331286 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:29.336052 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:29.372534 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:29.544785 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:29.574815 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:29.583640 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:29.590027 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:29.596153 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:29.603978 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:29.613313 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:29.818661 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:29.842710 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:29.869004 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:29.870613 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:29.877573 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:29.881944 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:04:29.895594 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:29.896751 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:29.923431 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:29.962070 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:04:29.987782 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:30.101670 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:30.111948 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:30.142972 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:30.148494 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:30.150597 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:30.153944 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:30.183388 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:30.275945 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:04:30.369737 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:30.419991 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:04:30.431948 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:30.440083 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:30.444047 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:30.461665 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:30.462337 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:30.478024 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:04:30.480552 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:30.482946 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:30.529213 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:30.542351 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:04:30.554839 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:30.589424 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:30.600545 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:30.626969 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:30.640811 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:30.657450 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:30.682617 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:30.691893 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:30.693089 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:30.698437 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:30.887955 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:30.911162 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:30.912197 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:30.926674 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:30.941227 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:30.966485 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:30.970194 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:30.989257 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 45099.2us
W20240501 14:04:31.027890 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:31.035538 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:31.137719 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:04:31.138904 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:31.150748 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:31.175987 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:31.181453 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.181991 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:31.201445 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.208364 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.224792 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:31.228335 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:31.244879 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:31.285667 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:31.306134 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:31.311585 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:31.439535 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:31.441763 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:31.471383 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:31.471695 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:31.483492 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.495127 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:31.501981 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.509917 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:31.539799 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:31.659826 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:31.766300 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:31.778365 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:31.796201 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.809921 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:31.817251 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:31.839443 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:31.840910 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.947721 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:31.948022 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:31.991983 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:31.998778 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.007952 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:32.021095 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.234724 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:32.246132 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:32.246573 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:32.257031 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.260517 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.266978 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:32.277714 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:32.349216 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:32.395642 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:32.463991 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:32.481073 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.490494 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:32.503612 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:32.530928 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:32.548662 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.592312 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:32.779547 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:32.826104 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.834384 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:32.839308 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:32.847761 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:04:32.856459 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:32.858292 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:32.859922 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:32.877036 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:32.915800 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:04:33.062301 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:33.097855 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:33.120416 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:33.133666 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:33.146883 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:33.161350 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:33.169337 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:33.170241 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:33.184144 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:04:33.196233 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.199932 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:33.201961 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:04:33.316413 17329 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:04:33.316620 17329 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:04:33.317400 17329 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:04:33.319658 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:04:33.321355 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:33.321462 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:04:33.322072 17329 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:04:33.337245 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:33.337188 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:33.350981 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.358670 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.373780 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.395018 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:33.417853 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:33.456250 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:33.532203 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:33.541834 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:33.567004 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:33.596092 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:33.625576 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:33.637902 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:33.644551 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.673830 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.675827 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:33.684929 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:33.699213 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:33.711937 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:33.731994 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:33.736796 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:33.828734 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:33.890280 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.910645 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:33.913426 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:33.915035 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:33.918251 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.934378 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:33.956895 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:33.990170 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:04:34.038172 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:34.047358 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:34.167546 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:34.167786 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:34.173090 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:34.187556 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:34.198864 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:34.201293 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:34.210342 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:34.463095 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:34.471614 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:34.498384 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:34.507310 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:34.527366 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:34.534379 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:34.549891 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:34.646826 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:34.716704 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:34.733551 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:34.750242 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:34.765614 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:34.765609 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:34.767485 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:34.768842 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:34.793577 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:34.882411 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(60b3db79ff754e8bbe9e5865f7104312): perf score=0.033384
I20240501 14:04:34.906433 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(2020fe37059a4e74bb45190398e4e177): perf score=0.033349
W20240501 14:04:35.002912 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:04:35.027123 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(60b3db79ff754e8bbe9e5865f7104312) complete. Timing: real 0.144s user 0.052s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":420,"dirs.queue_time_us":3675,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":6812,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":834,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21035,"threads_started":3,"wal-append.queue_time_us":16656}
I20240501 14:04:35.033409 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(60b3db79ff754e8bbe9e5865f7104312): 4921 bytes on disk
I20240501 14:04:35.034482 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(60b3db79ff754e8bbe9e5865f7104312) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 14:04:35.036216 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(86bfedcdabe44e7ab135674163935786): perf score=0.033349
W20240501 14:04:35.063746 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:04:35.147117 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(2020fe37059a4e74bb45190398e4e177) complete. Timing: real 0.239s user 0.071s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":12217,"dirs.queue_time_us":5639,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":4678,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1037,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":26578,"threads_started":3,"wal-append.queue_time_us":4129}
I20240501 14:04:35.150049 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(2020fe37059a4e74bb45190398e4e177): 4921 bytes on disk
I20240501 14:04:35.150993 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(2020fe37059a4e74bb45190398e4e177) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
I20240501 14:04:35.156540 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(86bfedcdabe44e7ab135674163935786) complete. Timing: real 0.120s user 0.041s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4438,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1464,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":943,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4702,"threads_started":1,"wal-append.queue_time_us":2736}
I20240501 14:04:35.158143 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(86bfedcdabe44e7ab135674163935786): 4834 bytes on disk
I20240501 14:04:35.159113 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(86bfedcdabe44e7ab135674163935786) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 14:04:35.186344 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:35.213629 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:35.236865 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:35.271855 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:35.282490 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:35.293596 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:35.322695 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:35.328604 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:35.328536 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 60465.2us
W20240501 14:04:35.330528 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:04:35.331737 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:35.332129 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:04:35.423137 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:35.451145 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:35.577299 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:35.633561 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:04:35.634516 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:35.658135 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:35.659456 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:35.666275 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:35.683825 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:35.687728 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:35.688532 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:35.704121 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:04:35.765617 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:35.913722 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(f516f01e242d4b8789130201f748f3ea): perf score=0.033354
W20240501 14:04:35.914283 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:04:35.914551 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:35.922533 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:36.031669 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(f516f01e242d4b8789130201f748f3ea) complete. Timing: real 0.118s user 0.063s sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10021,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":11150,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17534,"threads_started":2,"wal-append.queue_time_us":6878}
I20240501 14:04:36.033241 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(f516f01e242d4b8789130201f748f3ea): 4581 bytes on disk
I20240501 14:04:36.034370 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(f516f01e242d4b8789130201f748f3ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 14:04:36.145087 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:36.146246 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:04:36.148650 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:36.164023 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:36.167727 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:36.187831 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:36.196343 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:04:36.212615 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:36.222503 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:36.222782 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:36.262838 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:36.271217 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:04:36.319430 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:36.372386 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:36.398154 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:36.433460 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:36.450475 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:36.451865 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:36.454164 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:36.455231 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:36.456316 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:36.464144 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:36.514883 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:36.594725 16218 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:04:36.594930 16218 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:04:36.595814 16218 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:04:36.597011 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:04:36.598136 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:04:36.598407 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:04:36.598501 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:04:36.598755 16218 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:04:36.658733 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:36.665817 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:36.678501 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:36.679423 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:36.694221 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:36.715888 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:36.716454 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:36.751153 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:36.912065 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(efe7fed8e5d44e8a80ffbdc27619a74d): perf score=0.033356
I20240501 14:04:36.980266 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(efe7fed8e5d44e8a80ffbdc27619a74d) complete. Timing: real 0.068s user 0.043s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":409,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":6469,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1109,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":635,"threads_started":2,"wal-append.queue_time_us":409}
I20240501 14:04:36.982442 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(efe7fed8e5d44e8a80ffbdc27619a74d): 4834 bytes on disk
I20240501 14:04:36.983287 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(efe7fed8e5d44e8a80ffbdc27619a74d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
I20240501 14:04:37.063400 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:37.078969 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:37.079319 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:37.081693 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:37.092499 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:37.095086 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:37.145246 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:37.172961 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:37.202195 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:37.223632 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:37.231355 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:37.234331 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:37.244760 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:37.259750 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:37.271095 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:37.357929 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:04:37.468358 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:37.615434 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:37.796689 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(de161b10c84f4bcc85f8cf7b199f99db): perf score=0.033350
I20240501 14:04:37.809552 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:37.813480 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:37.829479 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:37.858812 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:37.876611 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:37.883757 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:37.895951 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:37.912894 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:37.929530 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:37.932390 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:37.936581 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:37.948138 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(de161b10c84f4bcc85f8cf7b199f99db) complete. Timing: real 0.151s user 0.048s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":26941,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":18243,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7450,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":30733,"threads_started":2,"wal-append.queue_time_us":3709}
I20240501 14:04:37.949646 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(de161b10c84f4bcc85f8cf7b199f99db): 4581 bytes on disk
I20240501 14:04:37.950457 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(de161b10c84f4bcc85f8cf7b199f99db) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240501 14:04:37.957999 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:37.977981 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:37.986035 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:37.999440 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:38.009291 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:38.023213 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:38.130216 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:38.134236 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:04:38.260663 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:38.263900 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:38.281265 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:38.282080 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:38.292304 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:38.303032 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:38.307003 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:38.315413 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:04:38.343233 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:38.381021 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:38.581921 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:38.602012 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:38.606328 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:38.621958 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:38.626472 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:38.645272 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:38.652709 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:38.656847 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:38.684806 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:38.688069 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:38.689224 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:38.694586 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:38.704821 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(e0bf08cbb6d347d783b1fa7762c802df): perf score=0.033342
I20240501 14:04:38.713207 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:38.741828 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(39f1ab3a94db41b08ac90bba52418a93): perf score=0.033353
I20240501 14:04:38.796658 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:38.813335 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:38.861111 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:38.873325 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:38.887060 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:38.897316 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:38.899168 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(e0bf08cbb6d347d783b1fa7762c802df) complete. Timing: real 0.194s user 0.051s sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41486,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":5880,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":886,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":59092,"threads_started":2,"wal-append.queue_time_us":15754}
I20240501 14:04:38.901877 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(e0bf08cbb6d347d783b1fa7762c802df): 4581 bytes on disk
I20240501 14:04:38.903632 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(e0bf08cbb6d347d783b1fa7762c802df) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
W20240501 14:04:38.913254 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:38.921047 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:38.927405 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:38.929380 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(39f1ab3a94db41b08ac90bba52418a93) complete. Timing: real 0.187s user 0.054s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":25417,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":31314,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":962,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":93330,"threads_started":2,"wal-append.queue_time_us":66010}
I20240501 14:04:38.931008 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93): 4834 bytes on disk
I20240501 14:04:38.931885 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240501 14:04:38.944279 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:39.130460 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:39.150501 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:39.158108 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f): perf score=0.033378
I20240501 14:04:39.199455 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.041s user 0.029s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1929,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":813,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":237,"threads_started":1,"wal-append.queue_time_us":369}
I20240501 14:04:39.201002 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f): 4921 bytes on disk
I20240501 14:04:39.201648 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240501 14:04:39.351014 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:39.356024 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:39.421010 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.435577 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.435691 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.525789 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:39.564313 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:39.639359 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:39.647439 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:39.657286 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:39.678759 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:39.684173 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.693387 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.694077 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:39.728586 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:39.757797 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:39.765568 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:39.937660 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.939642 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:39.945160 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:39.963936 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.965544 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:39.977644 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:40.129161 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:40.166972 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:04:40.179075 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:04:40.255880 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:40.267815 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:40.271373 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:40.308943 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:40.312973 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:40.316903 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:40.317745 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:04:40.333222 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:40.334172 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:04:40.339131 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:40.353782 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:40.358937 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:40.386839 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:40.391258 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:40.395045 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:40.419785 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:40.439857 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(ec1b3fec785f43c4a3bacedecd8472dd): perf score=0.033395
W20240501 14:04:40.477149 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:40.526024 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:40.564045 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(ec1b3fec785f43c4a3bacedecd8472dd) complete. Timing: real 0.124s user 0.041s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3152,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":5396,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":935,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3242,"threads_started":2,"wal-append.queue_time_us":309}
I20240501 14:04:40.565177 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(ec1b3fec785f43c4a3bacedecd8472dd): 4581 bytes on disk
I20240501 14:04:40.565770 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(ec1b3fec785f43c4a3bacedecd8472dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240501 14:04:40.567046 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(f4402de0b48041c383568feb9365147e): perf score=0.033374
W20240501 14:04:40.606127 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:40.616917 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(f4402de0b48041c383568feb9365147e) complete. Timing: real 0.050s user 0.044s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":79,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1407,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":202,"threads_started":1,"wal-append.queue_time_us":326}
I20240501 14:04:40.618180 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(1fdedd39e12a49659d989f37b3287d87): perf score=0.033373
I20240501 14:04:40.649690 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(1fdedd39e12a49659d989f37b3287d87) complete. Timing: real 0.031s user 0.024s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":221,"threads_started":1,"wal-append.queue_time_us":315}
I20240501 14:04:40.651218 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(1fdedd39e12a49659d989f37b3287d87): 4581 bytes on disk
I20240501 14:04:40.651829 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(1fdedd39e12a49659d989f37b3287d87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 14:04:40.653035 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(f4402de0b48041c383568feb9365147e): 4581 bytes on disk
I20240501 14:04:40.653641 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(f4402de0b48041c383568feb9365147e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 14:04:40.721338 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(7c0bc741ed064dd7ad8ec70219496c6f): perf score=0.033391
W20240501 14:04:40.758941 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:40.766940 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:04:40.773527 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(7c0bc741ed064dd7ad8ec70219496c6f) complete. Timing: real 0.052s user 0.034s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1465,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6896,"threads_started":2,"wal-append.queue_time_us":6456}
I20240501 14:04:40.775627 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(7c0bc741ed064dd7ad8ec70219496c6f): 4921 bytes on disk
I20240501 14:04:40.776226 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(7c0bc741ed064dd7ad8ec70219496c6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240501 14:04:40.828601 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:04:40.910543 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:40.985038 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:40.992812 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:40.999621 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:41.008033 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:41.026561 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:41.029284 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:41.034862 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:41.048177 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.048414 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.059695 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:41.076867 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:41.086329 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:41.099747 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:04:41.111838 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:41.133188 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:41.141614 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:41.162333 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:41.184968 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:04:41.191612 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:04:41.191913 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:41.288807 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:04:41.310420 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:41.354244 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:41.400714 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.413975 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.419643 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:41.425235 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.430914 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:41.446856 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:41.460206 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:41.468888 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:41.531347 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(ec1b3fec785f43c4a3bacedecd8472dd): perf score=0.033392
W20240501 14:04:41.537572 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:41.603477 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:41.630183 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(ec1b3fec785f43c4a3bacedecd8472dd) complete. Timing: real 0.099s user 0.064s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":597,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1842,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":994,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":499,"threads_started":2,"wal-append.queue_time_us":389}
I20240501 14:04:41.632037 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(ec1b3fec785f43c4a3bacedecd8472dd): 4581 bytes on disk
I20240501 14:04:41.632603 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(ec1b3fec785f43c4a3bacedecd8472dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240501 14:04:41.633509 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(6aaf52ee71b44511b7b8909711ab0a42): perf score=0.033379
I20240501 14:04:41.700673 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(6aaf52ee71b44511b7b8909711ab0a42) complete. Timing: real 0.067s user 0.041s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1554,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3262,"threads_started":1,"wal-append.queue_time_us":3145}
I20240501 14:04:41.702293 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(6aaf52ee71b44511b7b8909711ab0a42): 4834 bytes on disk
I20240501 14:04:41.702986 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(6aaf52ee71b44511b7b8909711ab0a42) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 14:04:41.729182 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:41.752353 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.756666 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.756743 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.759311 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:41.771113 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:41.967737 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:41.969918 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:41.989447 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.992614 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:41.993778 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:42.010885 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:42.042963 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:42.104154 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:42.162292 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f): perf score=0.033338
I20240501 14:04:42.198302 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.035s user 0.031s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":382,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1869,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":530,"threads_started":2,"wal-append.queue_time_us":391}
I20240501 14:04:42.200604 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f): 4921 bytes on disk
I20240501 14:04:42.201452 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(2998b60071cd408aa1cd06fb57d76a8f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240501 14:04:42.303905 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:42.313958 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:42.335614 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:42.338279 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:42.341497 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:42.349941 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:42.351317 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:42.369084 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:42.455308 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:04:42.584405 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:42.591874 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:42.601511 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:42.602945 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:42.619710 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:42.623911 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:42.649662 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:04:42.709264 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:42.725482 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:42.868609 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:42.871712 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:42.898393 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:42.901475 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:42.902132 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:42.929656 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:42.931947 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:42.950654 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:42.956504 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(bb712628c7764a138f5425d14b724d9e): perf score=0.033367
I20240501 14:04:42.967675 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(12510f71b6a742ab8cc359243de8119b): perf score=0.033343
I20240501 14:04:43.031410 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:43.070190 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(bb712628c7764a138f5425d14b724d9e) complete. Timing: real 0.113s user 0.046s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13502,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":2138,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":869,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13663,"threads_started":2,"wal-append.queue_time_us":396}
I20240501 14:04:43.072034 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(bb712628c7764a138f5425d14b724d9e): 4581 bytes on disk
I20240501 14:04:43.072667 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(bb712628c7764a138f5425d14b724d9e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 14:04:43.076515 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:43.096997 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:43.097368 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:43.102771 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:43.110687 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:43.112565 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:43.164181 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(12510f71b6a742ab8cc359243de8119b) complete. Timing: real 0.196s user 0.050s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":18612,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":935,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":36357,"threads_started":2,"wal-append.queue_time_us":6357}
I20240501 14:04:43.166317 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(12510f71b6a742ab8cc359243de8119b): 4921 bytes on disk
I20240501 14:04:43.179335 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(12510f71b6a742ab8cc359243de8119b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240501 14:04:43.271118 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:43.303763 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:04:43.313704 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:43.321561 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:04:43.322870 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:43.367250 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:43.392481 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:43.392539 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:43.411267 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:43.413885 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:43.420325 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:43.456379 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:43.504117 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:43.589758 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:43.625989 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:43.656877 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:43.665405 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:43.667385 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:43.668308 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:43.678556 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:43.686764 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(e8b9db597e654ed48b679abf58b8a2c8): perf score=0.033349
I20240501 14:04:43.693770 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:43.694500 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:43.696354 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:43.748143 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:43.761377 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:43.809754 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:43.813390 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(e8b9db597e654ed48b679abf58b8a2c8) complete. Timing: real 0.126s user 0.060s sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6898,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":47579,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":909,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10873,"threads_started":2,"wal-append.queue_time_us":3932}
I20240501 14:04:43.814895 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(e8b9db597e654ed48b679abf58b8a2c8): 4922 bytes on disk
I20240501 14:04:43.815451 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(e8b9db597e654ed48b679abf58b8a2c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240501 14:04:43.920550 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:43.929500 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:43.959885 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:43.966046 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:43.975082 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:43.980664 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:43.996963 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.021203 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.078703 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(6b6a54ac32694640bffdf1c7f3b035cc): perf score=0.033364
I20240501 14:04:44.126699 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(6b6a54ac32694640bffdf1c7f3b035cc) complete. Timing: real 0.048s user 0.038s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":371,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1516,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1974,"threads_started":2,"wal-append.queue_time_us":1843}
I20240501 14:04:44.128500 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(763b2e84afa64924ad5eeeeb3265a1d8): perf score=0.033371
W20240501 14:04:44.135246 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:44.142073 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:44.174995 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(763b2e84afa64924ad5eeeeb3265a1d8) complete. Timing: real 0.046s user 0.036s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":95,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1746,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":350,"threads_started":1,"wal-append.queue_time_us":499}
I20240501 14:04:44.176393 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(763b2e84afa64924ad5eeeeb3265a1d8): 4834 bytes on disk
I20240501 14:04:44.177000 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(763b2e84afa64924ad5eeeeb3265a1d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 14:04:44.178084 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(6b6a54ac32694640bffdf1c7f3b035cc): 4579 bytes on disk
I20240501 14:04:44.178769 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(6b6a54ac32694640bffdf1c7f3b035cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
W20240501 14:04:44.208678 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:44.262518 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:44.279196 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:44.306545 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.343226 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:44.359541 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.362846 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.511507 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:44.553331 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.560062 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:44.561555 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.568260 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.578730 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:44.593737 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:44.594687 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:44.650720 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:44.669643 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:44.683135 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(63b7f2a8d8a84faf86c4de16b3cc6663): perf score=0.033365
I20240501 14:04:44.743049 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(63b7f2a8d8a84faf86c4de16b3cc6663) complete. Timing: real 0.060s user 0.044s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":998,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":4416,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":718,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3582,"threads_started":2,"wal-append.queue_time_us":2822}
I20240501 14:04:44.746948 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(63b7f2a8d8a84faf86c4de16b3cc6663): 4834 bytes on disk
I20240501 14:04:44.747669 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(63b7f2a8d8a84faf86c4de16b3cc6663) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
W20240501 14:04:44.805681 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:44.841172 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:44.869905 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.874217 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.880031 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:44.881650 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:44.905974 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:44.989866 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:44.992087 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:04:45.032720 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:45.055413 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:45.072245 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:45.084666 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:45.097617 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:45.097715 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:45.103940 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:45.176090 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:04:45.252653 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(4d8c435795a04b1b90b6e2ff914676c8): perf score=0.033337
I20240501 14:04:45.292798 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(4d8c435795a04b1b90b6e2ff914676c8) complete. Timing: real 0.040s user 0.032s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":364,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1559,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":925,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":512,"threads_started":2,"wal-append.queue_time_us":362}
I20240501 14:04:45.293929 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(4d8c435795a04b1b90b6e2ff914676c8): 4834 bytes on disk
I20240501 14:04:45.294490 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(4d8c435795a04b1b90b6e2ff914676c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240501 14:04:45.391988 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:04:45.395800 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:04:45.398293 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:45.407222 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:45.422675 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:45.427772 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:45.441844 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:45.446966 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240501 14:04:45.473930 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:45.496047 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:45.514894 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:45.544590 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:45.561882 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:45.662165 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:45.681008 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:45.702598 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:45.702836 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:45.716596 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:45.722148 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:45.729259 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:45.752885 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:45.812139 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:04:45.827426 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(1fdedd39e12a49659d989f37b3287d87): perf score=0.033380
I20240501 14:04:45.903826 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(1fdedd39e12a49659d989f37b3287d87) complete. Timing: real 0.076s user 0.060s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3318,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1941,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":911,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7167,"threads_started":2,"wal-append.queue_time_us":3831}
I20240501 14:04:45.905582 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(1fdedd39e12a49659d989f37b3287d87): 4581 bytes on disk
W20240501 14:04:45.905731 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:45.906325 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(1fdedd39e12a49659d989f37b3287d87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240501 14:04:45.962373 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:45.966734 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:45.968286 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:45.988859 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:45.992841 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:45.996220 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:45.996770 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:45.997695 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:46.008080 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:46.099462 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:46.139457 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:46.167968 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:46.255975 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:46.285861 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:46.287194 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:46.298813 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:04:46.300956 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:46.315511 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:46.315945 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:46.345431 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:46.356757 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:46.371364 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:46.402834 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:46.418716 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:46.444034 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:46.456403 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:46.477157 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:46.492096 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:46.559221 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:46.604975 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:46.648085 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:46.822084 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:46.862339 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:46.870271 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:46.905524 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:46.913555 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:46.931288 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:46.967049 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:04:46.969565 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:47.038559 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:47.109201 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:47.133184 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:47.135004 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:47.154559 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:47.171233 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:47.180352 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:47.253011 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:47.268759 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:47.320495 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:47.329560 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:47.465060 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:47.467336 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:47.489501 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:47.506600 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:47.519666 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:47.526733 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:47.544045 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:47.588562 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:47.673699 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:47.682899 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:47.707978 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:47.768637 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:47.811636 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:47.864709 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:47.874917 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 44559us
W20240501 14:04:47.937230 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:47.950896 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:04:47.973697 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:48.021479 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:48.063196 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:48.073024 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.080335 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:48.090427 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.094448 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:48.104993 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:48.123229 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:48.177322 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:48.212486 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:48.314842 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:48.322871 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:48.340373 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:48.345362 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.347878 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.354501 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.373531 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:48.459292 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:48.595800 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:48.598423 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:48.601701 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.623486 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.625428 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:48.637019 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:48.641886 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:48.655107 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:48.703795 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:48.724212 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:48.737210 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:48.854442 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:48.865717 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:48.876093 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:48.881013 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:48.907819 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:48.916527 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:48.926972 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:48.928629 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:04:48.937719 19200 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:04:48.937916 19200 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:04:48.938599 19200 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:04:48.941246 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:48.943709 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:48.947754 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:48.947891 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:48.947948 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:04:48.948609 19200 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:04:48.969489 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.080726 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:49.081746 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:49.107532 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:49.112543 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.131177 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.155694 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:49.181713 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:49.194795 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:49.364995 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:49.381940 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:49.466958 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.467123 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:49.467576 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:49.511523 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.515045 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:49.582535 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.633940 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:49.640545 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:04:49.655975 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:49.687641 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:49.687956 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:49.709522 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.718582 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:49.727666 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:49.775279 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:49.803527 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:49.913066 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:50.103770 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:50.104588 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:50.117101 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:50.117764 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:50.122617 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:50.151547 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:50.155966 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:50.214888 19127 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:04:50.215111 19127 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:04:50.215868 19127 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:04:50.217123 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:04:50.218091 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:04:50.219166 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:50.219341 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:04:50.222092 19127 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:04:50.231817 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:04:50.365728 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:50.383065 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:50.390969 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:50.426400 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:50.439643 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:50.468003 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:04:50.469720 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:04:50.471848 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:50.486284 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:50.486425 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:50.492414 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:04:50.536533 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:50.556993 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:50.640736 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:50.643033 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:50.664726 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:50.689705 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:50.694262 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:50.715847 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:50.719794 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:50.790585 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:50.824458 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:50.875234 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:04:50.877384 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:50.942507 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:50.958971 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:50.961207 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:50.980530 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.029830 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.047871 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.050941 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:51.067694 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:51.112844 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:51.114744 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:51.143807 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.191980 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.209650 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:51.232628 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:51.244652 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:51.293639 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:51.294082 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:51.297554 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:51.309276 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:51.339342 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:51.353179 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:04:51.354148 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:51.415369 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:51.443553 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:51.487576 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:51.508672 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:51.522439 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.530138 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:51.550323 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.565358 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:51.653085 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:51.728950 19679 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:04:51.729157 19679 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:04:51.730051 19679 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:04:51.736366 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:51.745090 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:04:51.745342 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:04:51.745642 19679 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:04:51.822700 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:51.841064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.848901 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:51.852859 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:51.876957 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:51.922219 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:51.950619 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:51.957576 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:51.969846 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:52.019302 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:52.021169 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:52.028265 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.051980 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:52.052093 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:52.055518 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.056823 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:52.235643 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:04:52.322867 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:52.345690 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:52.351725 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.375542 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.388710 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:52.388851 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 55144.1us
W20240501 14:04:52.407864 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:52.410282 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:52.417439 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:52.491588 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:04:52.609836 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:52.610080 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.613997 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:52.629531 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:52.630841 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.649600 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:52.694228 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:04:52.827301 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:04:52.857780 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:52.868598 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:52.879199 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:52.879361 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:52.884387 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.910198 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:52.913713 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:53.018275 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:04:53.166275 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:53.170794 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:53.191506 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:53.192741 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:53.206138 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:53.261696 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:53.264907 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:53.325320 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:04:53.392345 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:53.393234 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:53.393247 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:53.436471 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:53.442252 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:53.442312 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:53.443219 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:53.459152 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:53.474751 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:53.517158 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:53.683883 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:53.700647 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:53.726761 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:53.739652 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:53.750346 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:53.756592 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:53.777315 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:53.790540 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:53.819036 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:53.851151 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:04:53.881839 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:53.942754 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:53.950033 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:53.956491 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:04:53.957027 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:53.964123 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:04:53.980294 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:53.981564 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:54.005807 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:54.010977 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:54.011984 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:54.013048 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:54.144081 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:04:54.205582 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(6feda9b53e484e7ea40a5890b7d51ac4): perf score=0.033334
W20240501 14:04:54.239989 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:04:54.252071 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:54.267405 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:54.280009 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:54.287297 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.341217 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.405061 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:54.405787 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:04:54.408192 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(6feda9b53e484e7ea40a5890b7d51ac4) complete. Timing: real 0.202s user 0.104s sys 0.057s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9278,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":6476,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":138496,"threads_started":2,"wal-append.queue_time_us":128984}
I20240501 14:04:54.410670 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4): 4921 bytes on disk
I20240501 14:04:54.412442 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":655,"lbm_reads_lt_1ms":4}
W20240501 14:04:54.420792 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:54.572352 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:54.580405 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:04:54.698804 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:54.732663 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:54.750895 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:54.754971 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:54.755674 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:04:54.758344 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.771849 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.777429 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.789357 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:54.806005 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:54.817900 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.850420 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.884341 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:54.900328 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:54.991010 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:04:55.012611 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:04:55.067066 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:55.166002 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:55.169535 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:55.182478 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:55.188794 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:55.202392 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:55.212421 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:04:55.213477 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:55.389454 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:04:55.447387 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:55.459786 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:55.469312 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:55.469540 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:55.476681 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:55.480870 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:04:55.484417 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:55.540902 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:04:55.571502 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:04:55.572690 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:04:55.683027 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:55.720345 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:55.723057 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:55.723961 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:55.740710 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:55.753355 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:55.766974 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:55.859436 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:04:55.870739 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:55.971998 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:04:56.023767 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:56.040747 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:56.040892 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:56.041729 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:04:56.044731 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:56.052278 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:56.052556 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:56.066170 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:56.085433 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:56.107928 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:56.152870 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:56.215570 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:04:56.223265 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:56.253718 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:56.255259 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:56.290546 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:56.294948 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:56.306910 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:56.321624 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:56.339586 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:56.380775 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:56.479458 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:04:56.481102 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:56.518291 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:04:56.533500 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:56.554170 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:56.573391 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:04:56.590580 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:56.592187 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:56.596391 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:56.637355 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:56.637501 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:56.653501 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:04:56.766806 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:56.771893 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:56.792860 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:56.811260 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:56.822579 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:56.834100 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:56.962853 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:57.006253 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:57.088687 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.091569 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:57.093801 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:57.102893 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:57.104002 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.108978 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:57.151455 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:57.161528 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:57.182894 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:04:57.318380 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:04:57.343904 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:04:57.398768 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.402055 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.423460 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:57.426491 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:57.433732 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:57.436764 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:57.549299 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:04:57.595507 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:57.608335 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:57.619697 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:04:57.623036 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:04:57.645115 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.658133 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.663800 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.685310 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:57.880990 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:57.890533 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:57.917496 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:57.994060 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.035598 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:58.040074 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:04:58.050550 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:58.051218 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:04:58.068054 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:58.219305 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:04:58.281307 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:58.288969 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.293007 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:58.295107 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.297008 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:58.327419 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:58.455374 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:58.485184 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:58.491434 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:58.532757 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:58.571832 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:58.587674 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:58.604175 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.613541 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.620544 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.625509 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:58.636575 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:58.689460 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 69811.1us
W20240501 14:04:58.691463 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:04:58.733310 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:04:58.758332 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:58.769223 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling FlushMRSOp(6aaf52ee71b44511b7b8909711ab0a42): perf score=0.033340
W20240501 14:04:58.776506 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:58.833791 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: FlushMRSOp(6aaf52ee71b44511b7b8909711ab0a42) complete. Timing: real 0.064s user 0.035s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6342,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1011,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":852,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12971,"threads_started":2,"wal-append.queue_time_us":6340}
I20240501 14:04:58.834810 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(6aaf52ee71b44511b7b8909711ab0a42): 4834 bytes on disk
I20240501 14:04:58.835427 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(6aaf52ee71b44511b7b8909711ab0a42) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240501 14:04:58.874996 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:58.921150 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:04:58.934597 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:58.941669 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:58.964530 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:58.974957 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.975890 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:58.979943 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:59.026944 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:04:59.097708 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:59.120606 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:59.122380 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.126477 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.129920 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:59.132236 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.133046 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:59.152760 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:04:59.240475 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:04:59.395691 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:59.418373 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:59.439164 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:59.443255 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.449920 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.452064 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:59.453155 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:04:59.477181 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:04:59.546660 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:04:59.557330 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:04:59.617591 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:04:59.691334 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.706228 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:59.708304 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.710064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:59.710693 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:59.716089 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:04:59.762907 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:04:59.847812 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:04:59.977604 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:00.000567 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.010982 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:00.017592 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:00.018360 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.036087 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:00.041301 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:00.063606 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:00.089319 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:05:00.225991 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:00.316515 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:00.334767 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:00.366510 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:05:00.367666 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.398941 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.429230 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:00.430475 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:05:00.450124 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.527503 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:00.549103 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:00.558909 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:00.561199 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:00.582082 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:00.584705 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:00.591444 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:05:00.598520 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:00.605914 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:00.616461 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:00.648516 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:05:00.819430 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:00.838775 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.846716 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.858342 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:00.875113 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:00.876921 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:00.877576 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:00.902549 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:00.929129 19681 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:05:00.929344 19681 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:05:00.930267 19681 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:05:00.931928 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:05:00.933409 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:05:00.936223 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:00.936349 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:05:00.936580 19681 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:00.999935 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:05:01.042097 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:05:01.046370 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:01.048430 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:05:01.054683 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:01.141005 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:01.153304 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:01.160948 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:01.161425 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:01.170322 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:01.172245 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:01.182926 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:01.211422 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:01.266206 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:01.289760 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:05:01.375175 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:01.424180 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:01.431695 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:01.439193 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:01.451911 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:01.454141 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:01.460820 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:01.476181 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:01.610934 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:05:01.612387 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:01.690109 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:01.700909 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:01.738495 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:01.746230 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:01.748292 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:01.785637 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:01.788558 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:01.799530 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:01.912746 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling FlushMRSOp(60b3db79ff754e8bbe9e5865f7104312): perf score=0.033383
W20240501 14:05:01.954644 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:01.976210 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: FlushMRSOp(60b3db79ff754e8bbe9e5865f7104312) complete. Timing: real 0.063s user 0.043s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":958,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1066,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1092,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":4352,"thread_start_us":1104,"threads_started":2,"wal-append.queue_time_us":416}
I20240501 14:05:01.977947 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(60b3db79ff754e8bbe9e5865f7104312): 4921 bytes on disk
I20240501 14:05:01.983978 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(60b3db79ff754e8bbe9e5865f7104312) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240501 14:05:02.001163 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:02.004055 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:02.012854 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:02.018187 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.021210 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.024266 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.043608 20324 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:05:02.043846 20324 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:05:02.044664 20324 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:05:02.050108 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:02.050953 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:02.051084 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:05:02.051556 20324 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:02.222311 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:05:02.239491 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:02.246004 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:02.246578 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:05:02.254781 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:02.352818 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:02.383911 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:02.415741 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:02.424366 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:02.446781 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.447315 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.448343 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:02.461975 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.470808 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:02.554987 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.566874 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:02.592620 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:02.593323 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:02.664929 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:02.779348 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:02.841257 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:02.841645 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:02.983790 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:05:03.069629 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:03.095100 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:03.104226 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:03.111675 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:03.121374 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.167410 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.172232 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:03.188007 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:05:03.232861 19681 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:05:03.233085 19681 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:05:03.233819 19681 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:05:03.234617 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:05:03.235795 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:05:03.242650 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:03.242834 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:05:03.243105 19681 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:05:03.278997 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:03.294267 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:05:03.294469 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:03.310562 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.315271 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.336589 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:03.379040 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:03.417296 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:05:03.455740 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:03.519099 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:03.545336 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:05:03.604115 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:03.608104 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:03.610992 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.612605 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:03.643750 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.670583 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:03.708186 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:05:03.764437 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:05:03.829283 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:05:03.917747 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:03.933673 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:03.949193 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:03.973625 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:03.985561 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.986166 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:03.999966 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:04.000841 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:04.004305 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:04.047971 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:04.060977 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:05:04.105108 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:04.145463 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:04.148929 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:04.156203 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:04.169550 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:04.170928 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:04.203157 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:05:04.219184 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:05:04.537626 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:04.542323 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:04.547669 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:04.571683 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:04.612993 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:04.652325 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:04.669931 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:04.684962 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:04.724123 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:05:04.807328 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:04.823156 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:04.844476 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:04.854970 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:04.865566 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:04.865561 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:04.878033 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:04.916132 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:04.984968 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:05.006525 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 14:05:05.047132 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:05:05.079735 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:05.113322 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:05.170526 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 51752.9us
I20240501 14:05:05.211529 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:05.239930 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:05.251871 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:05.255753 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.298486 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.299510 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:05.307725 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.435590 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:05.444272 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:05.456130 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.465749 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.468201 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:05.474965 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:05.485934 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:05:05.487968 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:05:05.512804 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:05.549268 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:05.592168 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:05:05.696144 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:05:05.699050 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:05:05.708575 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:05.726135 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:05.727003 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:05.743825 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:05.745101 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.749248 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.770897 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.913051 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:05.918599 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:05.921124 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:05.921743 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:05.926142 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:05.973551 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:05.983152 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:05.989127 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:06.002065 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:06.083359 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:05:06.116648 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:06.140630 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:05:06.235399 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:06.236851 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:06.260947 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:06.266669 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:06.273874 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:06.275606 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:06.308669 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:06.446822 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:06.569672 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:06.571786 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:06.606480 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:06.646615 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:06.649883 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:06.653116 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:06.659221 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:06.689006 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:06.693897 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:06.704849 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:06.729317 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:05:06.736459 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:05:06.740463 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:06.742332 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:06.758718 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:06.772701 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:06.784188 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:06.811672 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:06.826206 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:06.839305 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:06.844563 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:07.067124 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:07.080886 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:07.092523 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:07.099764 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:07.120995 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:07.131944 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:07.145051 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:07.307716 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:07.344025 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:07.369513 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:07.371098 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:07.404232 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:07.411890 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:07.424685 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:07.437834 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:07.547991 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:07.549703 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:07.579164 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:07.653278 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:07.679356 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:07.686391 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:07.686677 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:07.687497 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:07.691449 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:07.704703 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:07.732245 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:07.776490 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:07.999131 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:08.017585 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:08.039072 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:08.045811 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:08.048528 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:08.077762 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:08.094290 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:08.095626 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:08.134680 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:05:08.166846 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:08.174564 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:08.221562 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:08.221719 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:08.253898 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:08.262135 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:08.270104 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:08.362709 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:08.388033 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:08.421547 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:08.432348 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:08.464984 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:08.484721 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:08.499652 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:08.500048 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:08.550794 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:08.552057 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:08.571198 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:08.578251 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:08.772527 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:08.788816 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:08.796373 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:08.810403 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:08.811228 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:08.815300 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:08.838465 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:08.863227 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:08.999392 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:09.012950 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:09.070088 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:09.072548 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:09.074057 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:09.085537 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:09.095052 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:09.097353 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:09.108726 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:09.230849 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:09.305881 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:09.373502 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:09.382098 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:09.413221 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:09.422919 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:09.432689 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:09.470274 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:09.486500 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:09.519287 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:09.563519 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:09.569792 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:09.664206 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:09.676812 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:09.676807 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:09.683862 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:09.685300 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:09.698392 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:09.847026 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:09.857071 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:09.912820 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:09.924190 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:09.948664 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:09.955039 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:05:09.962554 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:09.965461 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:09.988907 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:10.037639 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:10.074424 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:10.145354 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:05:10.173710 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:10.179725 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:10.180590 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:10.192997 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:10.197049 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:10.214371 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:10.214809 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:10.285334 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:10.442518 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:10.457224 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:10.458845 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:10.463613 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:10.473724 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:10.475153 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:10.475957 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:10.495640 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:05:10.497385 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:05:10.531250 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:10.551524 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:10.600656 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:10.654767 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:10.701782 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:05:10.739188 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:10.758109 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:10.776947 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:10.784884 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:10.795308 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:10.814374 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:10.818423 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:05:10.834008 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:10.847115 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:10.943778 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:10.984365 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:11.014815 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:11.026186 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.026659 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:11.033986 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:11.048954 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:11.076761 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:11.078804 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.182004 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling FlushMRSOp(6feda9b53e484e7ea40a5890b7d51ac4): perf score=0.033396
W20240501 14:05:11.275493 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:05:11.279130 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:11.293766 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: FlushMRSOp(6feda9b53e484e7ea40a5890b7d51ac4) complete. Timing: real 0.111s user 0.065s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":641,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":2686,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1091,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7458,"threads_started":2,"wal-append.queue_time_us":6809}
I20240501 14:05:11.300030 30842 maintenance_manager.cc:419] P 8fc5d8222c164a679e7a99966da1eaaf: Scheduling UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4): 4921 bytes on disk
I20240501 14:05:11.300830 30732 maintenance_manager.cc:643] P 8fc5d8222c164a679e7a99966da1eaaf: UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240501 14:05:11.327966 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.333761 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:11.383267 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:11.400847 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.423518 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:11.423676 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:11.479601 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:11.506338 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:11.522214 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:11.581636 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.595206 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.605293 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.608211 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:11.649613 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:11.688114 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:11.707253 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:11.799221 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:11.883302 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:11.884174 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:11.890411 19681 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:05:11.890693 19681 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:05:11.891669 19681 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:05:11.892858 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:05:11.899706 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:05:11.910354 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:05:11.912271 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:11.912415 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:05:11.912757 19681 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:05:11.930209 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.932493 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:11.958277 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:05:11.963104 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:11.981467 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:12.016708 29073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 55969.1us
W20240501 14:05:12.021896 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:12.081176 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:12.102213 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:12.205775 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:12.232991 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:12.268823 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:12.277043 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:12.292325 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:12.308449 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:12.308431 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:12.336509 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:12.422572 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:12.498270 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:12.503616 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:12.513849 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:12.539894 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:12.541064 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:12.560338 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:12.567346 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:12.576143 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:12.736943 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:12.769721 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:12.779958 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:12.801172 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:12.812772 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:12.823551 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:12.823961 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:12.851656 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:12.874748 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:12.895485 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:12.933699 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:12.977339 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:13.034992 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:13.044610 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.048696 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:13.055125 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:13.063791 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.065079 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:13.082787 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:13.112360 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:05:13.126358 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:13.222527 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:13.298915 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:13.302243 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:13.311738 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.315310 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.331446 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:13.339766 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:13.344406 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:13.443701 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:13.479691 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:13.514117 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:13.570178 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:13.601019 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:13.631951 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:13.633095 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.657711 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.658556 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:13.679690 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.843350 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:13.858225 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:13.859616 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:13.861881 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.871080 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:13.883451 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:13.920405 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:14.060703 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:14.069973 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:14.131052 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:14.137605 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:14.167624 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:14.172525 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:14.178519 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:14.180034 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:14.215389 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:14.354097 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:14.357190 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:14.371481 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:14.371995 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:14.388981 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:14.392977 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:14.453680 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:14.495417 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:14.497699 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:14.524044 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:14.539129 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:14.701447 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:14.714689 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:14.722206 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:14.725299 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:14.742866 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:14.752676 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:14.752887 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:14.852473 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:14.913487 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:14.913623 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:14.935599 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:15.052733 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:15.082314 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:15.105432 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:15.136124 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:15.143617 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:15.150486 19681 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:05:15.150746 19681 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:05:15.151512 19681 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:05:15.152558 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:05:15.152585 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
I20240501 14:05:15.153466 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:15.154250 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:05:15.155830 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:15.155977 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:05:15.156266 19681 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:05:15.157331 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.158131 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:15.167356 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:15.174888 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.185106 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:15.199389 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.242957 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:15.260205 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.307173 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:15.335016 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:15.345511 22747 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:05:15.345711 22747 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:05:15.346377 22747 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:05:15.349129 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:05:15.355473 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:05:15.364935 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:15.365087 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:05:15.365603 22747 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:15.439854 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:15.451457 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:15.473982 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:15.530652 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:15.542074 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.563880 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:15.587633 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:15.600118 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:05:15.623500 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:05:15.643699 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.664932 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:15.704097 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:15.796700 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:15.820464 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:15.830860 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.843711 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:15.844645 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:15.852960 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240501 14:05:15.853471 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:15.871245 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:15.878015 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:05:15.906195 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:15.916759 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:15.950095 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:16.016507 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:16.065670 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:16.086026 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:16.099663 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:16.101421 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.102210 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.105705 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:16.117014 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:16.324518 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:16.353298 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:05:16.365665 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:16.404481 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:16.434882 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:16.448326 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:16.454540 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:16.470307 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:16.499675 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.515115 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.529053 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.539316 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.572556 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.573444 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:16.581369 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:16.755386 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:16.844988 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:16.850450 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.859622 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:16.869632 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:16.870092 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:16.884271 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:16.910993 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:16.933495 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:16.950860 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:17.016592 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:17.107259 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:17.109501 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:17.110101 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:17.140202 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:05:17.149436 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:17.153694 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:17.155124 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:17.244818 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:17.271055 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:17.316711 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:17.375903 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:17.398517 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:17.401669 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:05:17.419945 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:17.424150 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:17.453187 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:17.453555 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:17.467711 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:17.469686 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:17.493512 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:17.644738 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:17.709848 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:17.728323 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:17.735863 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:17.767364 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:17.778120 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:17.787204 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:17.822074 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:17.828928 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:17.934087 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:17.992538 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:18.003968 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:18.032269 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:18.054618 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:18.068877 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:18.072460 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:18.078723 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:18.116698 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:18.164343 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:18.246937 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:18.262027 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:05:18.284402 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:18.369802 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:18.384496 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:18.407276 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:18.417196 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:18.427235 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:18.481313 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:18.501699 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:18.523398 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:18.525846 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:18.539062 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:18.633863 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:18.640790 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:18.655548 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:18.681843 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:18.688243 28810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 77968.6us
I20240501 14:05:18.699851 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:18.702093 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:18.720860 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:18.980204 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:18.986502 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:19.010617 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.011693 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:19.011790 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:19.013283 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.018477 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:19.101557 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:19.273903 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:19.280421 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:19.288558 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:19.307525 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:19.330919 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.335353 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.336052 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:19.373658 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:19.384291 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:19.479077 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:19.482651 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:19.620990 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:19.621843 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:19.649785 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:19.652753 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.664830 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:19.673847 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.682775 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:19.731057 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:19.795837 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:19.799476 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:19.815186 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:19.820631 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:19.826217 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.863704 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:19.872337 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:19.886924 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:19.927063 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:19.930569 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:19.985108 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:20.120393 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:20.122520 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:20.124640 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:20.153113 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:20.159884 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:20.164964 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:05:20.175529 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:20.178978 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:20.214762 30730 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 49032.9us
W20240501 14:05:20.253568 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:20.305105 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling FlushMRSOp(2020fe37059a4e74bb45190398e4e177): perf score=0.033341
I20240501 14:05:20.336313 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: FlushMRSOp(2020fe37059a4e74bb45190398e4e177) complete. Timing: real 0.031s user 0.029s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2127,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":804,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":3328,"thread_start_us":2149,"threads_started":2,"wal-append.queue_time_us":349}
I20240501 14:05:20.337952 30974 maintenance_manager.cc:419] P a0e901103604414c829b40edf9f24c28: Scheduling UndoDeltaBlockGCOp(2020fe37059a4e74bb45190398e4e177): 4921 bytes on disk
I20240501 14:05:20.338541 30865 maintenance_manager.cc:643] P a0e901103604414c829b40edf9f24c28: UndoDeltaBlockGCOp(2020fe37059a4e74bb45190398e4e177) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 14:05:20.362458 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:20.403427 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:20.417035 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:20.421526 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:20.423038 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:20.437328 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:20.444815 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:20.457753 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:20.496978 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:20.552883 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:20.614346 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:05:20.636488 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:20.704519 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:20.709267 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:20.710325 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:20.721161 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:05:20.723749 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:20.731416 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:20.761467 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:20.803561 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:20.809871 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:20.813584 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:20.845872 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:05:20.976707 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:20.979799 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240501 14:05:20.995573 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:20.998245 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.019405 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.040153 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:21.040386 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:21.077452 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:21.116159 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:05:21.163697 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:21.192443 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.214807 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:21.215150 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:21.231657 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.240077 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:21.247571 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:21.253119 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:21.272755 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:21.393299 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 14:05:21.488757 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:21.490353 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:21.530967 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.539149 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.555243 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:21.557313 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:21.661177 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:21.673585 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:21.714097 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:21.724707 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:21.728412 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:21.740959 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.741300 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:21.742563 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:21.782442 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:21.921962 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:22.014456 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:22.015633 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:22.027083 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.041267 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:22.041739 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.052404 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:22.155437 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:22.179544 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:22.211547 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:05:22.212823 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:22.241734 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:22.248199 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:22.306358 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:22.343216 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:22.380034 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:22.395164 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:22.403514 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:22.420156 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:22.438958 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.453400 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:22.473083 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.620983 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:22.639382 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.644728 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:22.657773 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:22.658754 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:22.660454 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.677446 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:22.706354 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:22.716563 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:22.769088 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:22.831240 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:22.887311 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:22.888394 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:22.932091 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.959967 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:22.961477 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:22.975715 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:22.989116 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:23.019623 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:23.020228 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:23.149549 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:23.153724 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:23.159686 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.184523 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:23.188832 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:23.189899 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:23.209977 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:23.211185 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:05:23.213148 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:23.232641 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:05:23.263743 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:23.282583 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:23.426072 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:23.426187 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:23.428265 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:23.437621 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.438520 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.446295 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:23.544942 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:23.617882 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:23.709898 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:23.720567 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:23.727291 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:23.734508 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.743636 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.764418 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:23.778450 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.883476 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:23.898487 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.898965 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:23.915746 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:23.936606 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:23.936733 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:24.156615 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:24.158030 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:24.171794 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:24.207072 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:24.224244 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:24.232102 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:24.241885 24244 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:05:24.242130 24244 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:05:24.243178 24244 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:05:24.245069 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:05:24.248526 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:05:24.249177 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:24.249284 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:05:24.249549 24244 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:24.253357 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:24.260053 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:24.294075 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:24.425916 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:24.434324 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:24.443642 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:24.444514 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:24.466095 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:24.527546 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:24.614372 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:24.625962 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:24.658067 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:24.726392 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:24.732626 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:24.732928 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:24.745152 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:24.749658 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:24.753015 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:24.753031 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:24.763561 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:24.823526 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:24.847149 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:24.996206 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:25.006655 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:25.019702 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:25.021286 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:25.031010 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:25.055335 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:25.067309 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:25.095784 24352 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:05:25.096015 24352 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:05:25.096757 24352 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:05:25.100512 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:25.102442 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:25.111286 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:25.111632 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:05:25.112094 24352 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:25.169659 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:25.194517 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:25.206284 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:25.234206 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:25.245299 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:25.254580 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:25.270258 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(7c0bc741ed064dd7ad8ec70219496c6f): perf score=0.033394
I20240501 14:05:25.276087 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:25.305171 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:25.305238 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:05:25.310839 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:25.352429 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:25.362862 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(7c0bc741ed064dd7ad8ec70219496c6f) complete. Timing: real 0.092s user 0.050s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14725,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":875,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5153,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14902,"threads_started":2,"wal-append.queue_time_us":395}
I20240501 14:05:25.364199 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(7c0bc741ed064dd7ad8ec70219496c6f): 4921 bytes on disk
I20240501 14:05:25.365402 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(7c0bc741ed064dd7ad8ec70219496c6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240501 14:05:25.472652 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:25.521838 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:25.529042 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:25.537552 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:25.554194 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:25.581310 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:25.593919 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:05:25.594818 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:25.607753 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:25.642094 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:25.654166 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:25.659868 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:05:25.690502 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:25.723857 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:25.745410 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:25.759021 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:25.774773 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:25.778482 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:25.783329 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:25.799650 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:25.861263 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:25.932181 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:25.942771 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240501 14:05:26.037034 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:26.040550 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:26.048842 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:05:26.050889 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:26.052850 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.064638 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.082931 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:26.101264 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:26.189213 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:05:26.240106 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:26.371390 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:26.396876 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:26.400707 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:26.411813 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.436260 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:26.444625 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:26.462728 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 14:05:26.463924 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.589221 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:26.608279 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:26.610327 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:26.659127 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.673935 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.677501 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.706880 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:26.816342 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:26.834708 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:26.926738 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:26.947397 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.948139 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:26.956398 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:26.975733 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:26.980399 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:27.014616 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:27.089814 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:27.142079 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:27.142484 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:27.160898 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:27.174110 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:27.176174 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:27.177501 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:27.178005 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.178025 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.213892 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:27.282124 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:27.288283 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:27.290868 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:05:27.323904 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:27.417929 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:27.418437 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:27.447592 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:27.455559 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.470487 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:27.478945 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.494472 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:27.494494 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.760558 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:27.798983 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:27.811650 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.816908 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:27.824829 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:27.841643 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.842996 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.851842 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:27.859324 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:27.876749 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling FlushMRSOp(ec1b3fec785f43c4a3bacedecd8472dd): perf score=0.033340
I20240501 14:05:27.910344 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:27.916760 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:27.935324 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:05:27.940275 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:27.941872 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:27.964740 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:27.969621 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:05:27.995261 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.002493 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:28.014022 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.088807 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: FlushMRSOp(ec1b3fec785f43c4a3bacedecd8472dd) complete. Timing: real 0.211s user 0.068s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6610,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":1590,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6950,"threads_started":2,"wal-append.queue_time_us":1140}
I20240501 14:05:28.090169 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(ec1b3fec785f43c4a3bacedecd8472dd): 4581 bytes on disk
I20240501 14:05:28.090850 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(ec1b3fec785f43c4a3bacedecd8472dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240501 14:05:28.109370 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:28.174769 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:28.211280 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:28.282246 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:28.290947 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:28.308210 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:28.342422 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:28.378914 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.383666 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.387737 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:28.462783 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 83387.3us
W20240501 14:05:28.511729 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:28.579887 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:28.585629 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.599521 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:28.613318 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:28.626724 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.630874 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:28.633109 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:28.676317 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:28.702325 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:28.834847 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:28.837458 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:28.862138 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.866756 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:28.869748 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.878480 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:28.969671 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:28.975057 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:29.134771 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:29.142797 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.152132 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:29.164618 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.164745 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.166502 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:29.364022 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:29.370277 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:29.377453 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:29.422865 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:29.427745 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:29.433955 30730 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 49666.1us
I20240501 14:05:29.457736 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.459838 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:29.562978 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:29.572805 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:29.634497 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:29.637189 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:29.652150 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:29.664089 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:29.687618 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:29.702876 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:29.707275 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:29.718940 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.734071 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.773192 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:29.782807 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:29.809935 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:29.938055 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:29.947108 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:29.964041 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:29.965092 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.980974 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:29.983582 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.987862 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:30.017483 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:30.018790 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:30.094035 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:30.202407 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:30.214939 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:30.258224 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:30.259354 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.289033 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.290874 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:30.337965 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:30.376637 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:05:30.377816 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:30.462235 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:30.510922 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:30.531023 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:30.557813 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:30.562922 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:30.567126 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.573997 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:30.593328 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:30.609977 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:05:30.654317 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:30.664132 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:05:30.682840 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:30.738066 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:30.748869 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:30.776969 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:30.801484 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.826176 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.842164 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:30.846694 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:30.899591 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:31.029633 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:31.030928 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:31.040604 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:31.045974 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:31.048662 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:31.057529 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.060624 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:31.060619 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:05:31.076429 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:31.140959 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:31.152160 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:05:31.207239 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:05:31.269685 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:31.308993 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.312799 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:31.317656 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:31.333640 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.340514 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:31.355679 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:31.539048 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:05:31.596068 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:31.620327 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.625041 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:31.625367 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.625666 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:31.651225 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:31.654300 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:31.668746 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:31.794340 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:31.814100 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling FlushMRSOp(2020fe37059a4e74bb45190398e4e177): perf score=0.033334
I20240501 14:05:31.868186 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: FlushMRSOp(2020fe37059a4e74bb45190398e4e177) complete. Timing: real 0.054s user 0.031s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1003,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1376,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1101,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1328,"threads_started":2,"wal-append.queue_time_us":1092}
I20240501 14:05:31.869995 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(2020fe37059a4e74bb45190398e4e177): 4921 bytes on disk
I20240501 14:05:31.870848 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(2020fe37059a4e74bb45190398e4e177) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
W20240501 14:05:31.922269 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:31.979360 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:32.012796 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:32.019613 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:32.026789 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.055526 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.063302 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.123656 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:32.140966 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:32.162309 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:32.165185 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:32.170845 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:32.198226 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:32.204116 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:32.209220 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.210714 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.238461 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:32.299319 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:32.307888 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:32.317243 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:05:32.337865 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:32.457085 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:32.460765 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:32.461426 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:32.467334 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.499650 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.520663 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.563829 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:32.623615 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:32.639892 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:32.742022 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:32.752735 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:32.763712 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:32.783288 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:32.786926 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.792222 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.792915 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:32.793649 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:32.832852 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:32.902917 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:33.019835 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:33.031872 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:33.035830 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:33.050225 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:33.066440 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:33.072353 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:33.093430 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:33.106317 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:33.197152 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:33.262611 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:05:33.347047 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:05:33.354691 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:33.362339 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:33.372229 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:33.394330 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:33.411801 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:33.456660 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:33.464388 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:33.483209 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:33.500061 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:33.518334 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:33.544823 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:33.558434 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:33.582541 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:33.583169 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:33.638295 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:33.646085 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:33.672286 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:33.812111 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:33.872879 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:33.884487 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:33.885383 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:33.918114 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:33.921885 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:33.939566 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.117677 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:34.120929 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:34.128721 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.131247 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:34.138935 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.153740 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:34.155458 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:34.269508 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:34.333035 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:34.346616 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:34.407007 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:34.414654 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.424944 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:34.429855 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.433933 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:34.440727 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:34.511651 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:34.672472 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:34.672830 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:34.711802 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:34.730350 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.732093 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:34.781981 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:34.826325 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:34.826519 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:34.826612 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:34.828689 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.831583 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:34.839591 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:34.861336 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:34.980768 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.984314 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:34.996577 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:34.998606 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:34.999106 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:35.005476 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:35.073093 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:35.101217 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:35.217182 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:35.225150 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:35.242614 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:35.253918 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:35.255999 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:35.292819 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:35.316535 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:35.317531 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:35.356897 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:35.362933 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:35.505690 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:05:35.535558 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:35.549666 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:35.558351 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:35.559986 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:35.578048 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:35.586373 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:35.624754 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:35.738487 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:05:35.786789 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:05:35.825122 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:35.843298 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:35.844672 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:35.879669 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:35.882479 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:35.925853 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:35.932329 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:35.944940 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:35.968152 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:35.987746 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:36.024189 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:36.058049 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:36.089774 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:05:36.151154 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.151167 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:36.152312 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:36.173871 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:05:36.181149 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.186601 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:36.208341 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:36.213066 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:36.250865 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240501 14:05:36.367209 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:36.372542 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:36.379645 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.408567 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.413292 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:36.432952 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:36.534161 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:36.592634 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:05:36.614842 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:36.649951 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:36.685696 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:36.688127 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:36.705174 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.746663 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.780591 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:36.801688 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.835664 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:36.865201 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:36.882952 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.913148 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:36.934563 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:36.938088 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:36.940960 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.944234 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:36.966387 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 72601.3us
W20240501 14:05:37.087155 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:37.168376 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:37.236101 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:37.249917 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:37.261641 26258 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:05:37.261901 26258 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:05:37.262683 26258 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:05:37.263463 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:05:37.264171 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:05:37.265025 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:37.265229 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:05:37.265501 26258 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:37.281790 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:37.381845 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:37.442085 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:05:37.443029 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:37.460068 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:37.462821 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:37.478838 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:37.490146 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:37.499766 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:37.533074 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:37.588553 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:37.662325 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:37.685215 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:05:37.687582 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:37.695183 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:37.701784 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:37.705354 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:37.738854 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:37.749118 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:37.750149 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:37.810066 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:37.827708 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:37.926244 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:37.926399 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:37.944059 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:37.948565 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:37.950491 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:37.953508 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:37.976934 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:37.981848 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:38.053189 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:38.094906 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:38.104326 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:05:38.197880 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:38.222548 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:38.233394 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:38.246789 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:38.287823 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:38.313055 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:38.376698 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:38.388937 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:38.397779 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:38.406744 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:38.435257 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:38.445410 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:38.449725 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:38.452797 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:38.468340 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:38.510293 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:38.536042 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:38.544952 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:38.762850 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:38.767752 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:38.776149 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:38.790455 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:38.823735 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:38.872570 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:38.950512 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:38.989426 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:39.104290 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:39.108733 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:39.138362 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.155004 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:39.191534 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:39.195168 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:39.232507 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:39.233960 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:39.357697 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:39.366287 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.396266 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:39.398372 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:39.401196 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:39.404407 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.438548 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:39.487596 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:39.540439 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:39.568936 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:39.711419 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:39.718829 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:39.732041 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.732915 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:39.744473 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:39.753355 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:39.763285 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.771224 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:39.917723 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:39.943475 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:39.949157 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:39.949966 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:39.954469 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:39.976408 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.981137 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.986853 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:40.084589 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:40.102170 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:40.124536 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:40.174561 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:40.178902 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:40.197167 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.199474 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:40.202108 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:40.204372 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.213219 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:40.291045 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:40.325696 26366 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:05:40.325887 26366 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:05:40.326618 26366 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:05:40.328625 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:40.328787 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:40.328903 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:05:40.329169 26366 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:40.396907 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:05:40.470799 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:40.480341 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:40.482116 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:40.504725 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:40.509001 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.542651 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:40.543431 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:40.543956 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:40.568208 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:40.653563 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:05:40.655419 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:40.677057 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.715256 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:40.716326 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:40.717003 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.733657 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:40.846980 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:40.885212 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:05:40.935722 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:40.936285 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:05:40.944921 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:40.956697 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.964726 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:40.978590 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.984206 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:41.030313 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:41.048023 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:41.052871 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:41.110535 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:41.185582 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:05:41.193358 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:41.199940 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:41.207809 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:41.212471 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:41.231791 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:41.235397 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:41.245589 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:05:41.296614 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:41.383883 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:41.435334 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:05:41.480402 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:41.489064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:41.490958 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:41.492796 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:41.511308 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:41.525826 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:41.613046 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:41.646126 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:41.698226 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:05:41.793200 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:41.812987 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:41.813684 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:41.830515 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:41.835265 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:41.862438 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:41.915374 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:41.965883 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:41.988186 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:42.013525 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:42.018576 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:42.022755 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:42.040102 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.041769 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.048070 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.056620 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:42.147750 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:42.267189 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:42.344659 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:42.355823 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:42.377182 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.388742 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:42.389241 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.399024 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:42.401599 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:42.416235 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:42.528800 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:42.537168 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:42.562064 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:42.587669 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:42.588083 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:42.601747 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.620774 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:42.630775 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:42.638077 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:42.665679 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:42.707223 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:05:42.737919 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:42.804777 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:42.878976 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:42.881249 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.885149 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:42.894177 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:42.912698 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:42.928166 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:42.931761 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:42.947149 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:42.973801 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:43.020725 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:05:43.032176 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:43.177357 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:43.194017 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:43.206388 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:43.217938 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:43.250831 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:43.292825 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:43.301169 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:43.425921 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:43.431917 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:43.434346 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:43.443279 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:43.443667 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:43.448534 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:43.457464 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:43.462294 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:43.503723 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:43.587983 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:43.608889 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:43.699723 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:43.722841 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:43.727622 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:43.731922 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:43.770545 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:43.771286 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:43.799041 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:43.830809 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:43.930060 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:43.955196 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:43.993299 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.001935 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.005728 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:44.008623 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.040979 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:44.289121 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:44.292721 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:44.298980 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:44.306299 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:44.307329 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:44.327205 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:44.329361 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:44.340658 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.349547 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:44.516980 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:44.567677 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:44.615808 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:44.620488 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:44.627218 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.640579 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.693989 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:44.711678 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:44.784865 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:44.815910 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:44.818037 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:44.839376 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:44.854446 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.867053 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:44.874543 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:44.879107 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:44.896543 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:44.943859 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:44.977989 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:45.050730 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:45.070562 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:45.082495 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:45.085790 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:45.109700 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:45.111012 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:45.114312 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:45.114616 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:45.175961 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:45.196736 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:45.236097 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:45.344835 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:05:45.356671 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:45.362900 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:45.368361 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:45.395689 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:45.426137 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:45.438059 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:45.441022 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:45.459400 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:45.586563 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:45.659773 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:45.677635 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:45.679461 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:45.685268 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:45.688185 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:45.698879 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:45.719708 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:05:45.823009 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:45.891666 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:45.959656 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:45.976428 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:45.977039 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:46.003607 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:46.011368 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:46.074105 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 106952us
I20240501 14:05:46.111166 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.125903 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:46.140772 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.146299 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.163718 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:46.169663 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.211776 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.213789 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:46.231421 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:46.233459 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:46.247217 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:46.299500 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:05:46.328981 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:05:46.340626 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:05:46.455245 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:05:46.500878 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:46.501183 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:46.501256 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:46.508467 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:46.527595 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:46.528492 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.530228 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:46.536235 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:46.556216 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:46.725366 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:46.753132 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:46.755581 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:46.779255 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.783715 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:46.785472 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:46.796157 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:46.827371 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:05:46.959313 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:46.960108 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:47.012307 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:47.013679 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:47.020679 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:47.024883 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:47.034989 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:47.036892 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:47.053025 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:47.106050 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:47.196570 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:47.244508 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:47.315495 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:47.322479 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:47.343825 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:47.365617 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:47.366158 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:47.374859 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:47.388309 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:47.422701 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:47.475138 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:47.491072 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:47.544199 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:47.552039 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:05:47.572281 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:47.578392 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:47.593679 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:47.598711 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:47.650108 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:47.709853 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:47.816330 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:47.832562 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:05:47.834374 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:47.852473 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:47.879189 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:47.904654 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:47.933320 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:47.934675 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:47.964303 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:47.976611 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:48.035463 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:48.036406 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:48.062216 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:48.066417 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:48.077013 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:48.088199 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:48.093191 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:48.133788 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:48.159119 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:48.215674 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:48.372633 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:05:48.448084 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:48.450655 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:48.557588 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:48.616482 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:48.650107 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:48.691639 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:48.707085 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:48.738895 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:48.743054 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:48.753530 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:48.765180 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:48.766319 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:48.786036 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:48.817601 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:48.819736 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:48.828012 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:48.844437 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:48.883589 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:48.983107 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:49.066273 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:49.108131 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:49.111775 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:49.129285 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:49.131317 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:49.133930 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:49.138280 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:49.156211 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:49.179203 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:49.391814 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:49.456120 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:49.459491 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:49.491545 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:49.495872 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:49.498143 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:49.544492 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:49.598927 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:49.636793 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:49.654033 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:49.654896 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:49.694281 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:49.694938 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:49.697571 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:49.749444 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:49.774068 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:49.781615 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:49.842620 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:49.966398 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:50.017696 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:50.020500 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:50.036051 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:50.047915 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:50.051189 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:50.078847 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:50.099900 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:05:50.139690 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:50.309127 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:50.320741 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:50.343317 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:50.358026 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 69484.7us
W20240501 14:05:50.363332 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:50.387738 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:50.398927 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:50.411149 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:50.436533 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:50.449333 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:50.451499 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:50.452147 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:50.500815 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:50.540796 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:50.570675 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:50.575536 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:50.595963 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:50.614706 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:50.632468 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:50.645005 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:50.666110 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:50.681893 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:50.809074 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:50.882886 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:50.886936 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:50.892604 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:50.897768 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:50.915113 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:05:50.917322 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:50.923552 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:50.951017 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:51.045414 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:51.098764 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:51.122819 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:51.124212 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:51.137502 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:51.145596 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:51.145826 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:51.201126 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:51.321805 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:05:51.346120 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:05:51.430413 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:51.470891 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:05:51.472519 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:51.520735 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:51.524981 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:51.540243 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:51.579023 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:51.595660 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:51.613210 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:05:51.619187 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:51.624327 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:51.634337 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:51.650247 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:51.703043 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:51.721364 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:05:51.838941 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:51.850239 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:51.866690 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:51.879595 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:51.910315 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:51.919809 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:51.923219 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:51.971442 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:05:52.017130 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:05:52.082826 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:52.086431 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:52.095546 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:52.104139 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:52.107156 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:52.114109 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:52.304454 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:52.322873 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:52.382897 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:52.415853 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:52.434573 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:52.435365 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:05:52.447510 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:52.448177 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:52.476931 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:52.515535 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:52.536748 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:52.537009 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:52.559885 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:52.562263 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:52.597870 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:52.622361 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:52.690052 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:05:52.746340 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:52.807837 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:05:52.809562 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:52.831027 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:52.907137 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:05:52.908540 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:53.004703 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:05:53.096127 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:53.113907 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:53.125654 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:53.139741 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:53.171605 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:53.187814 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:53.210280 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:53.217425 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:53.218943 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:53.223219 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:53.240195 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:53.251057 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:53.293077 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:53.324831 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:53.327792 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:53.345746 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:53.420370 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:53.453125 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:05:53.476861 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:53.619678 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:53.624447 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:53.626354 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:53.626458 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:53.653342 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:53.701661 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:53.723956 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:53.745306 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:53.943964 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:53.972227 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:54.019531 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:05:54.031634 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:54.048928 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:54.055754 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:54.065151 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:54.069037 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:54.090677 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:05:54.098110 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:54.098524 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:54.099622 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:54.104840 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:54.120100 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:54.131310 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:54.144026 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:54.144757 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:54.158198 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:54.168151 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:05:54.405050 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:54.504763 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:54.742017 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:54.744702 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:54.768294 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:54.789342 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:54.822875 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:54.832921 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:05:54.833447 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:54.874644 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:05:54.886168 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:54.888401 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:54.893725 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:05:54.904431 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:54.916954 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:54.929162 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:54.943533 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:54.948395 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:55.005362 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:55.184670 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:55.185153 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:55.189127 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:55.198112 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:55.212924 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:05:55.217252 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:55.221164 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:55.367147 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:05:55.383498 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:05:55.446841 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:55.447376 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:55.532510 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:55.534121 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:55.546945 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:55.558251 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:55.558682 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:55.575239 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:55.594228 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:55.639513 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:55.655131 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:55.656512 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:05:55.677932 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:55.727882 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:55.729671 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:55.808720 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:55.825692 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:05:55.899219 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:55.917292 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:55.996227 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:05:55.996156 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:56.013990 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:56.115672 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:56.133299 27942 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:05:56.133524 27942 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:05:56.134290 27942 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:05:56.135936 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:56.141295 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:05:56.141458 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:05:56.141748 27942 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:56.165585 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:05:56.178056 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:56.182796 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:56.224349 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:56.238559 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:56.248200 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:56.297202 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:56.330376 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:05:56.341854 28647 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:05:56.342073 28647 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:05:56.343020 28647 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:05:56.344638 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:05:56.345007 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:05:56.347903 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:05:56.348402 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:05:56.348546 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:05:56.348855 28647 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:56.402990 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:05:56.443985 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:56.444416 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:56.445804 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:56.460597 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:56.464013 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:56.477407 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:56.481472 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:56.510955 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:56.516489 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:05:56.525497 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:56.640619 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:56.748140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:56.749864 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:56.753212 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:56.753649 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:56.803781 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:56.816540 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:05:56.818584 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:56.873380 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:56.990685 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:57.006608 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:57.017581 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.026063 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:57.039062 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:05:57.039237 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:57.041822 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:05:57.047582 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:57.058537 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 14:05:57.065953 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.226322 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:57.257364 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:57.260645 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:57.268504 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.268925 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.312089 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.312496 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:57.355031 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:05:57.375687 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:57.546602 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:05:57.577622 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:57.580101 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:57.589252 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.597448 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:57.599772 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:57.604574 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:05:57.617908 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:57.687330 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:05:57.788245 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:05:57.828183 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:05:57.864948 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:57.882984 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:57.892035 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:57.907222 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:57.911783 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:05:57.913214 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:57.914229 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.919111 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:57.924258 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:57.955605 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:58.135174 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:05:58.176671 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:58.207140 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:58.210239 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.210498 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:58.216558 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.249374 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:58.287721 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:58.324360 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:58.325659 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:58.334700 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:58.343691 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:05:58.360519 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 75690.7us
I20240501 14:05:58.370833 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.395020 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:58.402684 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:58.416448 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:58.469053 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:58.472848 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:58.632176 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:58.638278 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:58.652153 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.655990 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.663937 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:58.698421 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:58.752112 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:05:58.761834 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:05:58.825029 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:05:58.891003 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.895215 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:58.896880 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:58.903438 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.918685 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:58.935901 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:58.939617 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:59.006194 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:05:59.089783 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:59.136451 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:59.137619 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:59.143479 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:59.171738 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.173051 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.183509 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:59.199745 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:59.238364 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:05:59.332918 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:05:59.401721 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:05:59.506258 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:05:59.509532 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:59.524996 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.526032 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:59.528468 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:59.554342 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:59.554800 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.579361 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.613629 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:59.636083 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:59.637101 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.678368 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:59.701730 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.706084 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:05:59.759558 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:05:59.891507 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:05:59.922890 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:05:59.940563 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:05:59.941670 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:59.960454 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.972128 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.972509 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:59.983156 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:00.007845 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:00.063258 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:06:00.156673 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:06:00.165153 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:00.205847 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:00.207646 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:00.215042 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:00.251361 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:00.264487 28679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 45231.8us
I20240501 14:06:00.281298 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:00.284368 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:00.373919 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:00.476472 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:00.481945 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:00.487731 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:00.513373 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:00.516974 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:00.522042 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:00.523555 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:00.550894 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:00.642372 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:00.793706 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:06:00.815863 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:00.831383 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:00.890386 30730 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 71922.8us
I20240501 14:06:00.892127 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:00.907441 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:00.926029 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:00.952431 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:00.953097 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:00.953377 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:06:00.953661 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:00.968884 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:00.970515 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:00.989207 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:01.022078 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:01.042333 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:06:01.086666 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:01.113276 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:01.142557 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:01.411101 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:01.435022 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:01.435101 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:06:01.440402 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:01.441179 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:06:01.442143 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:01.478884 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:01.479928 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 14:06:01.494940 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:01.506532 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:01.512713 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:01.537238 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:01.632544 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:06:01.633646 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:01.636884 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:06:01.706719 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:01.707126 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:06:01.713269 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:01.739043 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:01.745780 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:01.748775 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:01.775571 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:01.842377 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:06:01.845238 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:01.846163 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:06:02.013877 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:02.018100 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:02.044234 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:02.044481 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:02.044952 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:02.048796 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:06:02.055902 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:02.123096 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:02.131385 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:06:02.325031 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:02.398133 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:02.418555 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:02.422821 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:02.433861 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:06:02.462679 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:06:02.467372 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:02.474988 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:02.516574 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:02.527731 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:02.536945 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:02.542611 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:06:02.544934 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:02.554083 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:06:02.563633 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:02.584139 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:02.589007 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:02.610695 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:02.640656 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:02.875845 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:02.888903 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:02.923709 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:06:02.927717 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:02.929482 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:02.939121 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:02.952327 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:02.952452 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:02.956722 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:02.963330 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:02.982698 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:03.060431 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:03.193007 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:06:03.199416 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:03.214274 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:03.231433 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:03.251791 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:03.257288 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:03.260645 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:03.267577 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:03.377557 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:03.431921 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:03.521375 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:03.528901 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:03.560745 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:03.561465 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:03.580492 30490 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:06:03.580718 30490 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:06:03.581492 30490 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:06:03.582594 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:06:03.584126 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:06:03.584851 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:03.584992 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:06:03.585835 30490 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:06:03.606474 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:03.627038 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:03.684036 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:03.688422 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:03.948397 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:03.958534 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:03.960175 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:03.968508 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:03.981771 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:06:03.993474 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:03.996503 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:03.998109 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:04.025422 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:04.069577 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:04.077597 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:04.082353 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.087033 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.089838 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:04.104980 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:04.146667 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:04.154075 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:04.169302 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:04.256400 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:06:04.303989 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:04.348783 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:04.359776 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:06:04.439893 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:04.446120 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:04.447126 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.456172 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.468300 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.472180 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:04.600306 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:06:04.718369 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:04.724778 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:04.726708 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.728994 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.731135 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:04.744427 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:04.788252 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:04.923558 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:05.000634 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:05.003958 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:05.014673 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:05.026976 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:05.035797 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:05.039005 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:05.096477 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:05.116289 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:05.121335 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:05.177770 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:05.187361 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:05.352995 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:05.357417 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:05.359794 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:05.361832 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:05.422838 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:05.449138 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:05.454914 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:05.483496 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:05.519518 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:05.589020 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:05.624063 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:05.631785 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:05.643833 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:05.644636 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:05.650167 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:05.661542 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:05.669534 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:05.744506 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:05.793100 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:05.910255 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:05.941668 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:05.967622 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:05.984172 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:06.000310 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:06.033061 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:06.034837 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.067092 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:06.089568 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.126462 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.215900 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.219476 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:06.221370 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:06.227169 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:06.270962 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:06.299409 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:06.364141 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:06.562913 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:06:06.564051 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:06:06.565425 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:06:06.612444 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:06.619889 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:06.674759 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:06.683173 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:06.695995 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:06.703836 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.705389 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.716370 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.721513 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:06.789222 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:06.818603 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:06.884135 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:06.884600 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:06.884745 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:06.889593 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:06.918327 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 14:06:06.919504 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:06.919947 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:06.929466 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:06.935024 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:07.033068 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:06:07.172531 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:07.186146 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:07.219409 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:07.225579 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:07.238763 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:07.242256 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:07.259498 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:07.280171 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:06:07.357434 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:07.412896 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:07.497157 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:06:07.512889 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:07.532298 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:07.544430 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:07.544936 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:07.597388 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:07.623240 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:07.637301 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:07.660343 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:07.806492 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:07.835470 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:07.846921 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:07.858191 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:07.873189 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:07.883682 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:07.891113 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:07.892556 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:07.903534 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:07.944929 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:07.981819 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:06:08.077840 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:08.098934 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:08.106559 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:08.123667 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:08.127893 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.138217 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:08.146108 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:08.147187 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:08.152508 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:08.182237 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:08.194572 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:08.313649 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:08.346895 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:08.435637 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:08.446017 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.451514 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.451740 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:08.453893 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.482867 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:08.504434 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:08.533090 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:08.733292 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:08.734222 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:08.744248 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:08.774461 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:08.779775 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:08.817363 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.822080 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.826153 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:08.832926 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:08.851825 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:06:08.870684 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:08.881045 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:08.907339 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:08.929333 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.929742 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.970592 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:08.979321 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:09.195000 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:09.211977 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:09.218091 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:09.225284 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:09.226661 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:09.230993 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:09.265645 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:09.321806 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:09.327937 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:09.346328 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:09.511973 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:09.514674 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:09.532189 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:09.534601 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:09.541144 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:09.554561 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:09.569320 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:09.610877 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:09.611240 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:09.620013 31382 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:06:09.620249 31382 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:06:09.620954 31382 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:06:09.622334 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:06:09.627130 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:06:09.628175 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:09.628469 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:06:09.628770 31382 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:09.657555 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:09.786232 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:09.827349 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:09.827498 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:09.837337 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:09.849273 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:09.860183 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:09.861686 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:10.052810 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:06:10.166697 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:10.180393 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:10.188199 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:10.196799 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:10.237788 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:10.247740 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:10.257874 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:10.271759 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:10.278975 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:10.285563 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:10.300730 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:10.320816 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:10.333236 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:10.336278 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:10.358989 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:10.368482 30728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.130:37325, user_credentials={real_user=slave}} blocked reactor thread for 155607us
I20240501 14:06:10.387151 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:10.401551 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:10.402947 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:10.442482 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:10.560900 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:10.566785 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:10.655650 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:10.660437 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:10.672049 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:10.678139 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:10.686384 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:10.687994 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:10.700029 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:10.763921 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:10.786720 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:10.898406 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:10.909705 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:10.916428 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:10.929289 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:10.933331 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:10.947111 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:11.047667 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:11.170437 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:11.184160 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:11.190344 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:11.194272 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:11.212426 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:11.215294 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:11.232292 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:11.233610 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:11.259027 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:06:11.309074 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:11.508076 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:11.594841 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:06:11.596036 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:11.624780 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:11.629622 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:11.641680 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:11.644024 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:11.657040 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:06:11.673795 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:11.688542 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:11.757233 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:11.775487 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:11.800586 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:11.802682 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:11.852604 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:11.884579 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:11.890722 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:11.920991 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:06:11.922175 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:11.924180 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:11.945119 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:12.011746 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:12.077373 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:12.077464 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:12.081451 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:12.113665 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:12.116477 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:12.127981 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:12.138434 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:12.264690 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:12.264777 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:12.293298 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:06:12.352483 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:12.356652 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:12.359546 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:12.383176 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:12.388513 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:12.390786 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:12.402997 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:12.457612 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:12.637634 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:12.651813 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:06:12.653095 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:12.662981 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:12.687911 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:12.693075 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:12.707767 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:12.715428 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:12.728276 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:12.740288 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:12.825086 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:12.863104 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:06:12.946997 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:12.965960 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:13.006667 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:13.017199 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:13.019503 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:13.047901 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:06:13.052137 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:13.060173 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:13.084239 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:13.136512 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:13.140882 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:13.156486 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:13.175782 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:13.190140 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:13.208482 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:13.218611 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:13.239121 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:13.257880 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:13.258311 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:13.282912 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 93532.9us
W20240501 14:06:13.348388 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:13.477198 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:13.531970 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:13.547502 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:13.557965 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:13.576786 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:13.577621 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:13.634313 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:13.641137 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:13.811045 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:13.817560 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:06:13.842403 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:13.846776 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:13.850780 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:13.859618 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:13.860805 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:13.881775 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:13.885272 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:13.965167 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:14.109794 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:14.110132 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:14.122169 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:14.127786 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:14.144493 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:14.153273 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:14.159364 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:14.289039 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:14.450789 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:14.458421 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:14.460775 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:14.462797 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:14.478264 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:14.489084 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:14.490502 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:14.503409 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:14.576295 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:06:14.597103 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:14.603317 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:14.639325 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:14.670711 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:14.676733 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:14.678882 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:14.691865 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:14.692262 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:14.721535 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:14.751695 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:14.790593 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:14.909129 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:15.045821 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:15.067831 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:15.085172 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:15.113236 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:15.115262 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:15.129748 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:15.156457 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:15.157940 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:15.188084 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:15.188863 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:15.194545 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:15.206218 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:15.212384 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:15.214426 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:15.221410 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:15.228900 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:15.359278 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:15.364557 31785 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:06:15.364809 31785 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:06:15.365787 31785 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:06:15.366775 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:06:15.367421 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:06:15.367691 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:06:15.367799 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:06:15.368086 31785 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:15.579716 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:15.583636 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:15.590133 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:15.611866 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:15.643118 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:15.764245 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:15.796778 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:15.847116 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:15.864241 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:15.864745 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:15.888464 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:15.916376 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:15.922381 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:15.931210 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:15.966301 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:15.976042 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:16.028210 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:16.047222 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.074942 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.075357 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.128557 32095 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:06:16.128769 32095 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:06:16.129580 32095 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:06:16.133190 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:16.135010 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:16.135160 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:06:16.135458 32095 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:16.189949 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:16.243613 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:06:16.314486 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:16.322935 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:16.329780 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:16.336709 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.346297 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.347416 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:16.356745 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:16.385361 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:16.556764 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:16.612176 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 14:06:16.684532 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:16.712524 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:16.722543 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.734246 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:16.754225 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:16.754660 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:16.756405 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.777330 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:16.787865 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240501 14:06:16.837934 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:16.858107 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.879824 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:16.891695 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:16.896310 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:16.920358 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:16.969144 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:16.991962 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:17.043747 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:06:17.044677 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:17.091073 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:17.147325 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:17.148792 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:17.155306 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:17.315254 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:17.372042 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:06:17.410430 31785 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:06:17.410688 31785 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:06:17.411669 31785 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:06:17.415691 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:06:17.416491 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:06:17.416734 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:17.416862 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:06:17.417853 31785 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:17.435459 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:17.443228 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:17.445725 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:17.454969 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:17.479403 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:17.499963 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:17.572389 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:17.582442 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:17.619269 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:17.653659 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:17.666194 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:17.677244 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:17.717832 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:17.850951 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:17.879627 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:17.921612 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:17.999756 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:18.021342 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:18.026437 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:18.037796 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:18.042963 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:18.045579 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:06:18.051038 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:18.053687 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:06:18.061985 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:18.113715 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:18.126022 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:18.126410 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:18.133740 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:06:18.199345 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:18.349081 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:18.353286 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:18.356177 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:18.360332 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:18.373561 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:18.398010 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:18.406715 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:18.407348 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:18.485288 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:18.515653 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:18.522403 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:18.543285 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:18.553421 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:18.579365 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:18.596099 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:18.600365 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:18.647464 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:18.728523 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:06:18.830971 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:18.888696 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:18.896059 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:18.904358 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:18.924721 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:18.925426 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:18.927367 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:18.928349 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:19.146696 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:19.172574 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:19.178375 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:19.182138 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:19.187783 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:19.188117 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:19.199398 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:19.212853 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:19.350416 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:19.431633 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:19.501822 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:19.513465 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:19.523728 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:19.534142 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:19.559080 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:19.564847 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:19.574353 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:19.577253 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:19.580333 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:19.584388 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:19.616118 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:19.641975 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:19.672109 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:19.672950 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 111423us
I20240501 14:06:19.679906 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:19.703186 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:19.721649 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:19.724174 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:19.733944 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:19.832020 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:19.911201 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:19.994858 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:19.995893 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:19.998922 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:20.010658 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:20.046403 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:20.083786 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:20.116123 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:20.302067 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:20.302059 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:20.329675 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:06:20.331174 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:20.332126 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:20.332149 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:20.366943 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:20.381238 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:20.426903 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:20.525465 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:20.570036 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:20.605265 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:20.615411 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:20.618098 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:20.619798 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:20.634188 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:20.645660 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:20.653512 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:20.654757 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:20.773561 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:20.855621 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:20.922765 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:20.932065 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:20.933508 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:20.954217 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:20.979480 29072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 78596.3us
I20240501 14:06:20.984004 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:20.986428 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:21.006508 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:21.013690 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:21.019299 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.078286 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.095511 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:21.130450 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.140185 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:21.156185 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:21.293609 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:21.341518 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:21.352224 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:06:21.399365 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:21.413295 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:21.434768 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:21.436395 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.441047 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.444669 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:21.445598 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:21.521827 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:21.679919 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:21.685516 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:21.686098 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:21.690974 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.697033 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:21.697407 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.706275 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:21.812378 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:06:21.813899 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:21.840566 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:21.852051 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:21.853026 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:21.863752 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:06:21.902666 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:21.940675 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:21.952086 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:21.952900 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:21.954829 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.975224 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.979161 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:21.980939 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:22.050321 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:22.082590 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:06:22.083624 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:06:22.227355 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:22.254390 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:22.269075 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:22.276680 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:22.303098 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.319969 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.378830 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:22.382676 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:22.394174 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:22.444607 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:22.461548 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:22.536114 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.537895 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.539325 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.543466 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:22.545034 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:06:22.638470 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:22.700454 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:22.766039 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:22.780547 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:22.781841 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:22.797716 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:22.811950 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.814159 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.817430 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:22.833066 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:22.944936 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:23.068336 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:23.077890 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:23.078248 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.085366 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:23.088271 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:23.100050 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.104889 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:23.150204 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:23.267473 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:23.312987 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:23.320946 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:23.332468 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:06:23.337268 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:23.352804 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:23.359711 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:23.376332 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:23.396443 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:06:23.400216 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:23.402539 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.419536 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.423677 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:23.640533 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:23.644284 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:23.646921 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.670114 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:23.675172 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.689522 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:23.714857 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:23.771672 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:23.784806 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:23.874423 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:06:23.890826 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:23.893886 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:23.914840 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.917260 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:23.920019 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:23.923979 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:24.043627 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:24.092375 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:24.147596 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:24.224086 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:24.235183 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:24.238332 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:24.246336 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:24.247282 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:24.258275 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:24.292484 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:24.304947 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:24.351275 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:24.374702 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:24.388115 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:24.400647 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:24.414422 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:24.419663 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:24.432799 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:24.516261 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:24.661427 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:24.674665 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:24.724195 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:24.774166 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:24.776362 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:24.802364 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:24.879273 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:24.903990 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:24.937184 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:24.959640 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:24.981715 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.026882 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:25.049607 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:25.054497 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:25.058440 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:25.075315 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:25.111034 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.115432 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:25.135761 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.141673 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:25.224261 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:06:25.294277 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:25.390972 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:25.412072 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:25.413810 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:25.442257 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.446547 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.452237 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:25.471086 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:25.477941 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:25.516075 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:25.548740 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:25.566119 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:25.725185 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:25.728587 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:25.737609 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:25.747751 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:25.760809 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:25.761396 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.791729 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.951736 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:25.953225 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:25.954596 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:25.968068 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:25.985834 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:25.986763 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:26.016492 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:26.038166 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:26.128085 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:26.143122 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:26.227440 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:26.234441 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:26.271683 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:06:26.272756 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:26.276259 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:26.291611 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:26.299880 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:26.315546 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:26.509421 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:26.527417 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:26.535490 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:26.542400 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:26.552347 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:26.571250 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:26.575681 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:26.576292 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:26.645882 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:26.692975 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:26.850454 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:26.852316 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:26.861658 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:26.883550 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:26.889935 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:26.900148 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:06:26.904949 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:26.906570 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:26.953058 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:26.976698 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:26.990343 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:06:27.002939 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 73121.2us
W20240501 14:06:27.056244 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:27.119228 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:27.129635 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:27.134356 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:27.142058 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:27.149108 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:27.156085 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:27.161257 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:27.169662 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:27.283578 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:06:27.303341 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:27.321939 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:06:27.323397 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:27.427203 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:27.448618 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:27.453233 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:27.480512 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:27.484947 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:27.506592 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:27.508718 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:27.613058 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:27.669740 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:27.691629 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:27.705209 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:27.706557 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:27.725972 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:27.726430 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:27.734316 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:06:27.736238 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:27.744118 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:27.999944 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:28.001094 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:28.012135 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:28.015501 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:28.018952 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:28.024832 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:28.028034 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:28.033490 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:28.112181 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:28.117195 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:28.295994 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:28.390079 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:28.401101 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:28.425025 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:28.427846 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:28.443298 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:28.452641 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:28.464318 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:28.481765 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:28.482823 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:28.484050 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:28.515929 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:28.520843 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:28.534199 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:06:28.535318 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:28.574759 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:28.604920 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:28.622330 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:28.652025 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:06:28.727689 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:28.735239 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:06:28.824788 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:28.835091 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:28.841559 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:28.843258 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:28.867987 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:28.882072 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:28.930861 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:28.952005 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:28.963996 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:29.103276 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:29.117504 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.125878 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.126164 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:29.128845 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:29.130189 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:06:29.135152 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:29.149415 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:29.244561 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:29.309864 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:29.341526 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:06:29.414079 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:29.419034 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:29.419158 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:29.420629 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.421481 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.427040 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:29.548904 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:29.551224 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:29.663960 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:29.715658 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:29.717509 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:29.735725 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:29.753523 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:29.775990 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:29.780614 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.805518 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.847226 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.854673 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:29.915226 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:29.931223 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:29.946339 1754 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:06:29.946560 1754 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:06:29.947432 1754 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:06:29.949717 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:29.951297 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:06:29.962184 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:29.962447 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:06:29.962812 1754 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:06:29.971378 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.973001 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:29.982960 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:29.990653 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:30.165254 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:30.167788 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:30.226171 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:30.232038 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:30.247289 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:30.273329 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:30.288066 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:30.354928 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:30.355484 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:30.454156 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:30.520615 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:30.535255 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:30.543934 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:30.558092 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:30.593158 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:30.608146 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:30.629742 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:30.742650 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:30.744221 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:06:30.802800 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:30.815482 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:30.829651 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:30.830106 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:30.834335 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:30.842720 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:30.864841 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:30.868958 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:31.046535 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:06:31.083220 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:31.123659 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:31.126359 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.127489 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:31.149962 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.171968 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:31.176162 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:31.199316 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:31.247117 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:31.270766 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:31.307693 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:31.342609 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:31.342958 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:31.405275 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.408638 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.411474 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:31.412098 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:31.423103 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:31.483554 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:31.677400 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:31.681432 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:31.685469 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.690913 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:31.701624 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:31.714102 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.715216 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:31.743846 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:31.815296 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:31.825377 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:31.928591 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:31.955415 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:06:31.957226 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:31.959611 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:31.964366 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:31.965541 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.966081 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:31.969722 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:31.975481 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:32.011631 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:32.104498 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:32.120051 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:32.136062 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:06:32.182206 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:32.202749 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:32.206249 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:32.207134 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.209757 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.246381 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:32.271761 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:32.375514 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:06:32.448791 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:06:32.484350 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:32.560611 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:32.582231 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:06:32.597528 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.599748 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:32.610697 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.641929 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:32.668921 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.683172 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.703539 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.705286 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:32.721184 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:32.738802 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:06:32.739892 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:32.743786 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:32.757673 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:32.779165 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:32.837417 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:33.053586 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:33.076447 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:33.091436 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:33.092900 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:33.100510 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:33.115849 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:33.119439 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:33.164916 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:33.346844 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:33.353523 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:33.364708 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:33.375640 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:33.376368 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:33.381542 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:33.384354 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:33.491999 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:33.507876 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:33.621842 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:33.635651 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:33.638718 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:33.640394 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:33.651151 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:33.656212 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:06:33.667121 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:33.687580 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:33.691385 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:06:33.696240 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:33.748775 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:33.783064 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:33.921671 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:33.972517 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:33.990569 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:33.991605 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:34.004029 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:34.013016 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:34.050470 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:34.054198 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:34.073065 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:34.099205 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:06:34.168195 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:34.174672 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:34.185135 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:34.223079 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:34.243611 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:34.261770 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:34.322808 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:34.327210 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:34.358129 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:34.442377 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:34.469051 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:34.499208 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:34.505511 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:34.506420 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:34.511147 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:34.512181 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:34.535190 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:34.540691 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:34.555694 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:34.595803 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:34.617383 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:34.751356 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:34.761303 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:34.762105 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:34.768735 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:34.793505 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:34.804245 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:34.811664 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:34.973413 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:35.041158 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.046316 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:35.050554 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:35.069249 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.072119 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:35.075089 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:35.089257 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:35.117568 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:35.125757 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:06:35.190518 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:35.213451 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:35.367748 2674 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:06:35.367990 2674 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:06:35.369083 2674 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:06:35.370740 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:06:35.372087 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:06:35.373443 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:35.373610 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:06:35.374368 2674 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:06:35.379739 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.387956 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:35.394785 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.401095 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.410100 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:35.427500 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:35.572229 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:35.601445 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:35.623663 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.624605 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:35.627981 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:35.641819 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:35.651137 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:35.659196 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.664155 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.943552 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:35.943799 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:35.952426 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:35.969951 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:35.970398 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:36.021907 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:36.045706 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:36.082002 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:36.100167 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:36.101888 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:36.122601 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:36.122993 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:36.125897 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:36.135669 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:06:36.136592 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:36.156570 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:36.191007 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:36.215695 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:36.297355 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:36.553210 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:36.565974 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:36.584331 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:36.606295 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:36.624611 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:36.673493 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:36.681648 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:36.697018 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:36.711066 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:36.732964 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:36.781904 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:36.835501 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:36.836118 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:36.858808 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:36.862058 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:36.873340 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:36.896692 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:36.899673 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:36.901607 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:36.945873 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:36.961733 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:06:36.962783 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:36.987370 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:37.089020 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:37.103541 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:37.109031 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:37.116155 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:37.128182 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:37.130084 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:37.139233 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:37.143047 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:37.217550 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:37.230494 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240501 14:06:37.306854 2610 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:06:37.307159 2610 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:06:37.308087 2610 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:06:37.310434 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:37.311793 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:37.311921 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:06:37.312187 2610 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:37.332069 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:37.421308 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:37.428920 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:37.460023 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:37.464447 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 46 similar messages]
I20240501 14:06:37.485396 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:37.523365 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:37.554646 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:37.587253 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:06:37.629530 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:37.641482 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:37.664600 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:37.665119 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:37.671787 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:37.717968 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:37.719012 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:37.750955 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:37.810590 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:37.901624 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:06:37.902701 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:38.139590 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:38.155283 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:38.155841 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:38.182972 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:38.195144 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:38.196096 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:38.204200 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:38.258013 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:38.350000 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:06:38.361146 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:38.405306 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:38.439383 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:38.456621 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:38.465261 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:38.485548 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:38.526329 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:38.560789 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:38.579861 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:38.634582 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:38.640622 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:38.667057 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:38.702009 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:38.728765 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:06:38.756335 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:38.760293 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:38.761041 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:38.783744 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:38.785645 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:38.803858 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:39.041453 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:39.058636 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:39.077301 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:39.082434 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:06:39.108553 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:39.109462 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:39.125836 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:39.130337 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:39.132355 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:39.137703 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:39.150830 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:39.191952 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:39.195695 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:39.256300 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:39.261672 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:39.292486 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:39.299702 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:39.306339 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:39.319031 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:39.415818 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:39.422433 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:06:39.476135 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:39.485340 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:39.544193 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:39.684139 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:39.689106 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:39.777902 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:39.844463 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:39.874944 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:39.924118 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:39.940150 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:39.946715 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:39.982694 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:40.005487 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:40.017117 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.031896 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.067030 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:40.069669 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:40.084906 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:40.090418 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:40.160398 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:06:40.165192 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 48721us
I20240501 14:06:40.172468 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:40.367416 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:40.383816 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.387840 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:40.393425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:40.399111 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:40.416674 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:40.441684 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:40.444217 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.472079 2674 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:06:40.472271 2674 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:06:40.472985 2674 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:06:40.473920 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:06:40.474509 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:06:40.474798 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:06:40.474900 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:06:40.475129 2674 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:40.585837 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:40.608430 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:40.671485 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:40.753530 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.766000 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:40.768841 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.785028 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:40.797675 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:40.819901 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:06:40.836428 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.861766 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:40.930552 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:40.931089 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:40.954571 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:40.990782 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:41.022359 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:41.034109 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:41.038378 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:41.049062 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:41.122507 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:41.176904 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:41.243417 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:41.292470 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:41.451400 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:41.452769 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:41.458159 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:41.463449 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:41.477699 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:41.477988 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:41.485548 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:41.603629 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:41.650696 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:41.663827 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:41.678046 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:41.684115 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:41.692667 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:41.701848 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:41.726307 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:41.726996 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:41.727982 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:41.773759 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:41.779124 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:41.790814 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:41.830413 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:42.048837 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:42.076397 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:42.079252 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:42.087369 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:42.099736 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:06:42.100831 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:42.102907 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:42.118358 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:42.127759 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:42.128052 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:42.128304 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:42.156347 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:42.158740 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:42.188088 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:42.206718 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:42.251492 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:42.310405 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:06:42.414577 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:42.462051 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:42.491590 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:42.498054 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:06:42.507272 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:42.523882 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:42.527101 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:42.547897 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:42.563593 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:42.652544 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:06:42.765678 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:42.767452 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:42.771806 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:42.780373 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:42.790609 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:42.793648 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:42.805729 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:42.815994 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:42.975461 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:06:42.976524 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:43.002502 3567 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:06:43.002755 3567 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:06:43.003571 3567 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:06:43.007052 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:43.009624 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:43.009871 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:06:43.010232 3567 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:06:43.029356 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:43.042073 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:43.045830 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:43.052630 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:43.071592 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:43.083354 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:43.155987 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:06:43.190940 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:43.325912 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:43.334803 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:43.342897 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:43.370942 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:43.376269 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:43.426647 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:43.427757 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:43.427791 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:43.452968 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:43.522312 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:43.584216 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:43.589552 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:43.605033 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:43.605835 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:43.615250 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:43.620330 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:43.656526 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:43.662806 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:43.800944 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:43.915508 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:43.920375 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:06:43.921617 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:43.922987 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:43.929983 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:43.940884 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:43.959198 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:43.964394 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:43.969318 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:44.008692 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:44.083169 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:44.156988 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:44.171590 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:44.181313 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:44.187788 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:44.194110 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:44.200785 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:44.201332 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:44.202383 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:44.289110 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:44.323316 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:44.341176 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:44.377631 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:44.466176 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:44.477422 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:44.483230 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:44.499461 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:44.512567 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:44.519665 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:44.539629 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:44.656807 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:44.663595 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:44.678689 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:44.777740 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:44.801329 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:44.802973 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:44.825251 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:44.834120 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:44.846028 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:44.883628 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:44.912505 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:06:44.920451 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:44.972652 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:44.998592 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:45.021660 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:06:45.027560 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:45.029551 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.037232 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.041606 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:45.229208 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:45.250296 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:45.259841 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:45.268079 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.279273 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.293586 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.317414 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:45.381632 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:45.459604 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:45.542083 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:45.547576 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.556005 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:45.568722 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:45.572573 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:06:45.580726 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:45.601613 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:45.629022 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:45.725580 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:45.815469 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:45.847388 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:45.861496 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.876278 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.878612 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:45.896787 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:45.908013 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:45.924666 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:46.005515 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:46.037796 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:46.046438 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:46.111307 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:46.175832 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:46.179805 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.187513 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:46.191680 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.213313 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:46.213541 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:46.262885 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:46.307288 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:46.316773 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:46.370797 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:46.390915 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.395670 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.396014 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:46.405825 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:46.409332 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:46.448048 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:46.512902 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:46.673144 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:46.679473 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:46.689776 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:46.691150 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:46.700790 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:46.706250 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.708110 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:46.719429 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:46.719799 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:46.724689 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:46.734714 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:46.890002 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:46.912597 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:46.916332 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.932579 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.940361 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:46.944679 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:46.956712 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:47.115976 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:06:47.116865 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:47.215934 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:47.235697 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:47.237097 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:47.244102 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:47.244765 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:47.260634 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:47.307768 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:47.321717 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:47.323235 4247 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:06:47.323427 4247 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:06:47.324110 4247 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:06:47.325901 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:06:47.326661 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:06:47.326826 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:06:47.326901 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:06:47.327332 4247 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:47.329432 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:47.356060 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:06:47.357286 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:47.412449 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:06:47.595299 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:47.599602 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:47.614571 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:47.628643 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:47.630802 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:47.655855 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:47.660056 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:47.660287 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:06:47.782891 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:47.816749 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:47.820888 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:47.824725 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:47.831377 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:47.856724 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:47.913066 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:47.933061 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:47.983275 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:06:48.188755 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:48.203500 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:48.205557 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:48.206785 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:48.223111 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:48.235561 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:48.261058 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:06:48.278236 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:48.285239 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:48.289682 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:48.312034 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:48.323626 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:48.370430 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:48.403535 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:48.438611 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:48.442159 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:48.459187 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:48.464408 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:48.466238 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:48.581670 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:48.705823 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:48.740671 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:48.885910 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:48.987826 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:48.998028 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:49.004169 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:49.020865 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:49.087937 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.104643 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.105667 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:49.128772 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:49.144587 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:06:49.145642 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:49.163583 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:49.167371 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:49.194319 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:49.211714 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:49.263965 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:49.270064 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.331758 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:49.332201 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:49.337064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.354471 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:49.433243 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:49.566265 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:49.570657 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:49.594211 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.623911 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:49.693435 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.693560 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.696238 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:49.714036 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:49.719954 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:49.751677 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:49.783727 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:49.891646 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:49.907326 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:49.938011 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.946204 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:49.958828 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:49.966099 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:50.001775 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:50.021725 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:06:50.114931 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:06:50.144697 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:50.147159 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:50.155917 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:50.167330 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:50.170970 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:50.177939 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:50.182436 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:50.182782 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:50.209120 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:50.442224 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:50.442945 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:50.452383 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:50.458457 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:50.478870 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:50.534684 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:50.625288 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:50.682618 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:50.699378 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:50.704541 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:50.706379 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:50.709910 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:50.737860 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:50.739212 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:50.742555 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:06:50.774933 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:50.792665 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:50.869959 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:51.061913 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:51.085451 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:51.085880 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:51.105803 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:51.118813 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:51.132302 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:51.149809 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:51.152228 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:51.182870 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:51.183008 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:51.210204 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:51.239825 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:51.261281 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:51.267621 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:51.319358 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:51.325227 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:51.336796 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:51.401742 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:51.574522 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:51.586355 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:51.603166 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:51.610747 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:51.613207 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:51.635648 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:51.639959 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:51.724575 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:06:51.746322 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:51.799726 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:51.818704 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:51.859990 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:51.865949 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:51.873557 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:51.881024 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:51.888082 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:51.892122 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:51.892120 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:52.032267 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:52.047493 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:52.062068 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:52.073840 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:06:52.127758 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:52.142731 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:52.143927 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:52.151288 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:52.163754 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:52.167865 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:06:52.177402 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:52.183504 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:52.204870 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:06:52.336983 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:52.495934 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:06:52.504429 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:52.507329 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:52.527482 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:52.533679 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:52.556651 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:52.560317 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:52.574959 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:06:52.612211 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:52.626169 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:52.641244 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:52.643745 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:52.701200 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:52.702247 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:06:52.702443 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:52.727816 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:52.788205 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:52.789094 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:52.796763 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:52.936937 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:52.951763 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:52.963222 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:52.974036 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:52.980523 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:52.989581 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:53.061420 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:53.201095 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:53.206610 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:53.212021 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:53.214255 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:53.216275 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:53.223989 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:53.225739 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:06:53.229053 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:53.243600 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:53.248965 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:53.261087 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:06:53.433696 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:53.472743 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:53.494572 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:53.495460 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:53.497839 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:53.507072 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:53.508049 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:53.520670 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:53.566668 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:06:53.740669 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:53.852319 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:53.875696 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:53.882889 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:53.883749 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:53.891201 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:53.892640 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:53.899156 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:53.903072 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:53.924448 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:06:53.931325 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:53.962618 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:54.022996 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:54.027401 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:54.033754 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:54.055500 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:54.129557 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:54.233111 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:54.247562 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:54.253433 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:54.272909 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:06:54.274334 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:06:54.297982 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:54.326391 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:54.349117 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:54.371570 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:54.434845 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:54.455248 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:06:54.465951 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:06:54.488420 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:54.498726 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:06:54.553627 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:54.554924 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:54.561323 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:54.588574 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:54.588968 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:54.604418 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:54.615439 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:54.623057 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:54.854770 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:54.861083 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:54.869457 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:54.873814 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:54.874276 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:54.876806 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:54.878917 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:54.961953 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:06:55.008057 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:55.075958 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:06:55.159073 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:55.182304 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.185050 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:55.203253 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.210688 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.215356 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:55.273401 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:55.317656 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:55.321815 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:55.387269 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:55.393380 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:06:55.398510 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.400667 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:55.402487 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.417851 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:55.431216 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:55.486521 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:55.572484 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:06:55.685169 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:06:55.701057 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:55.702016 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:55.711112 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:55.717550 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.743337 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:55.766079 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:55.775175 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:55.803830 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:55.804234 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:55.869604 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:06:55.910380 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:55.931758 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:55.959419 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.960140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:55.983489 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:55.995478 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:56.024976 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:56.075592 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:56.315538 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:06:56.321080 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:56.341082 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:56.376457 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:56.383051 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:06:56.408314 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:56.445665 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:56.474802 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:56.542013 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:56.692052 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:56.700021 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:06:56.710430 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:56.745851 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:56.749919 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:56.792801 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:56.802639 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:56.819844 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:56.848080 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:56.874974 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:56.929384 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:06:56.960549 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:06:56.969152 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:56.985802 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:57.010296 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.013151 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.019192 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.021505 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:57.032272 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:06:57.149796 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:57.189046 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:06:57.223695 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:06:57.249177 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:06:57.251425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:57.299257 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:57.331924 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:57.361974 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.362344 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.396118 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:57.411301 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:57.419166 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:57.427436 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:57.521811 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:57.562755 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:57.569602 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:06:57.599604 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:57.605428 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.611963 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:57.666909 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:57.674912 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:06:57.712230 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:06:57.829964 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:06:57.876751 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:57.878037 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.904307 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:57.913080 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:57.914134 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:57.929287 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:57.936815 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:57.962435 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:06:57.963505 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:58.005784 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:58.007318 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:58.172832 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:06:58.203207 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:06:58.206408 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:06:58.208664 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:58.226714 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:06:58.230070 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:58.243263 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:58.335331 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:58.343417 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:58.344676 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:58.396869 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:58.415978 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:58.468106 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:58.480512 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:58.514317 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:58.536625 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:58.576364 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:58.578172 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:58.619906 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:06:58.735036 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:58.751533 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:06:58.821192 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:58.852955 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:58.857252 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:58.861312 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:58.868348 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:58.889161 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:58.922111 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:59.144073 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:59.148602 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:59.167337 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:59.188809 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:59.194957 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:59.225430 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:59.328560 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:59.357496 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:06:59.366143 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:06:59.367301 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:06:59.462523 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:59.465332 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:06:59.476888 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:59.479599 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:06:59.480948 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:06:59.488245 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:59.500335 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:06:59.505472 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:59.513096 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:59.594440 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:06:59.622793 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:06:59.696992 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:06:59.768069 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:06:59.779392 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:06:59.809418 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:06:59.809942 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:59.813802 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:06:59.814334 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:06:59.863054 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:06:59.915951 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:06:59.937541 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:07:00.008091 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:00.014948 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:00.019053 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:00.022359 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:00.027753 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:00.034837 5851 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:07:00.035053 5851 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:07:00.035811 5851 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:07:00.038501 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:00.059001 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:00.062677 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:07:00.062945 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:07:00.063285 5851 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:07:00.083596 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:00.252349 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:00.336199 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:00.346895 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:00.356786 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:00.375255 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:00.400257 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:00.414469 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:00.440856 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:00.464000 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:00.475669 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:00.479321 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:00.511539 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:00.594318 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:00.608053 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:07:00.612543 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:00.631480 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:00.639845 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:00.641346 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:00.646836 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:00.675087 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:00.691345 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:00.850067 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:07:00.917966 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:00.926822 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:00.936288 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:00.976107 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:07:00.976979 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:00.981875 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:00.997908 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:01.017629 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:01.131623 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:01.131088 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:07:01.275720 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:01.279224 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:01.317440 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:01.345686 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:01.369887 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:01.388017 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:01.437426 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:01.497166 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:01.505540 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:01.524892 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:01.536613 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:01.556241 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:01.558364 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:01.561798 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:01.565266 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:01.709708 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:01.751184 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:01.807296 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:01.820317 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:01.845024 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:01.876194 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:01.877491 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:01.887729 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:01.902204 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:01.911310 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:01.938987 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:01.995245 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:07:02.073874 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:02.077507 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:02.081079 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.098995 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.106114 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.110499 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:02.133507 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:02.134737 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:02.141479 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:02.165482 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:02.224133 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:07:02.297950 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:02.340061 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:02.369406 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.375260 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:02.383536 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.387024 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:02.387166 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:02.412919 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:02.524420 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:02.600250 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 14:07:02.688657 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:02.692014 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:02.708207 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.708644 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.729874 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:02.731307 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:07:02.731686 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:02.805425 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:02.816910 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:02.822199 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:07:02.860088 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:02.870740 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:02.883751 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.885000 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:02.907855 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:02.926885 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:02.943035 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:02.999656 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:07:03.000494 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:03.009979 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:07:03.133690 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:03.165894 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:03.171808 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:03.177817 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:03.195745 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:03.200382 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:03.200385 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:03.233644 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:03.263363 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:07:03.420876 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:07:03.476819 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:03.502435 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:03.563114 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:03.584548 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:03.605952 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:03.612911 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:03.616680 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:03.628803 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:03.635406 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:03.862437 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:03.864243 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:03.869814 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:03.870818 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:03.923095 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:03.952984 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:03.963521 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:04.000039 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:04.087544 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:04.200105 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:04.212224 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:04.229069 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:04.241990 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:07:04.259876 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:04.274523 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:04.280213 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:04.302212 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:04.310700 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:04.390225 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:04.403510 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:07:04.405447 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:04.405874 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:04.443001 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:04.448685 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:04.454730 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:04.461336 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:04.462683 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:04.468279 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:04.509493 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:04.558552 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:04.738919 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:04.739624 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:04.772012 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:04.779482 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:04.786537 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:04.791379 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:04.801752 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:04.859197 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:07:04.919914 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:04.975152 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:04.985927 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:05.000020 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:05.009353 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:05.018889 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.044772 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.045485 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:05.070346 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:05.228130 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:05.240736 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:07:05.312310 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:05.316527 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:05.333005 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.351058 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:05.364696 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:05.369035 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:05.379806 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:05.414530 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:05.422082 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:05.440197 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:05.549577 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:05.646235 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:05.662817 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:05.730958 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:05.742147 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:05.746085 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.748477 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.756417 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.782608 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:05.801276 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:07:05.803750 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:05.845734 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:05.876268 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:05.881175 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.906932 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:05.909742 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:05.953801 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:06.167759 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:07:06.187520 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:06.188997 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:06.213785 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:07:06.228407 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:06.235944 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:06.243942 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:06.277735 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:06.295418 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:06.347626 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:06.435204 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:06.497504 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:06.509238 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:06.514243 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:06.545094 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:06.545285 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:06.613747 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:06.642076 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:06.747498 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:06.755420 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:06.760687 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:06.766069 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:06.769093 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:06.777516 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:06.782781 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:06.796561 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:06.809613 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:06.819702 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:06.839570 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:06.911088 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:06.983582 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 45361.9us
W20240501 14:07:06.999521 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:07.000023 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:07:07.079502 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:07.099109 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:07.101495 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:07.110360 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:07.129168 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:07.188515 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:07:07.210239 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:07.210703 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:07.219233 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:07.238817 6543 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:07:07.239017 6543 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:07:07.239892 6543 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:07:07.240732 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:07:07.241477 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:07:07.241873 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:07.244838 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:07:07.244985 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:07:07.245265 6543 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:07:07.249012 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:07.294564 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:07.299793 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:07.329674 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:07.368302 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:07.413363 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:07:07.439530 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:07.524744 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:07.537734 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:07.538800 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:07.552722 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:07.568543 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:07.575376 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:07:07.577349 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:07.664217 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:07:07.735610 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:07:07.817811 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:07.831971 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:07.832244 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:07.840850 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:07.863348 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:07.864660 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:07.865849 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:07.888401 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:07.899183 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:07.942256 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:07.985738 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:08.030690 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:08.041249 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:08.056478 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:08.059669 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:08.111676 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:08.124083 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:08.126593 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:08.140085 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:08.149531 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:08.171238 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:08.338207 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:07:08.339373 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:08.353451 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:08.493575 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:08.524714 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:08.544353 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:08.573415 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:08.602974 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:08.633080 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:08.639688 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:08.657269 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:08.690690 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:08.692767 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:08.693356 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:08.700049 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:08.703157 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:08.750178 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:08.912204 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:07:08.943401 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:09.007050 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:09.010841 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:09.019907 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.046694 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:09.073510 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.080929 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:09.140146 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:09.239848 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:09.245656 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:09.319780 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:09.354142 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:07:09.357756 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.370097 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.371196 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:09.383352 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:09.396301 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:09.402815 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:09.419059 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:09.445832 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:09.565958 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:07:09.566954 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:09.641548 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.642102 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:09.647918 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:09.657994 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.661106 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.670908 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:09.747233 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:09.769351 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:09.791178 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:09.926865 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:09.927071 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:09.932730 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:09.951467 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.970069 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:09.976945 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:10.001579 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:10.026480 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:10.158922 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:10.162083 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:10.163090 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:10.182989 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:10.192972 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:10.197335 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:10.215546 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:10.252074 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:10.291711 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:10.327316 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:10.472223 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:10.476186 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:10.517529 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:10.540098 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:10.566606 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:10.572546 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:10.586714 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:10.597260 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:10.597622 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:10.598024 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:10.607838 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:10.618359 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:10.627120 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:10.630532 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:10.658959 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:10.670547 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:10.691296 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:10.744259 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:10.759061 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:11.010476 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:11.153987 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:11.157480 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:11.167084 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:11.263080 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:11.276919 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:11.312912 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:11.320004 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:11.327402 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:11.390221 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:07:11.403162 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:11.405265 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:11.425627 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:11.441453 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:11.441985 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:11.448729 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:11.460791 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:11.490886 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:07:11.565609 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:11.653829 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:11.655072 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:11.753870 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:11.787156 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:11.787861 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:11.800733 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:11.803025 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:11.806722 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:11.808826 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:11.869488 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:11.912812 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:11.954531 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:11.996579 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:12.116258 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:12.117714 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:12.153720 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:12.155692 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:12.175359 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:12.186462 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:12.191829 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:12.227635 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:12.248284 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:12.251565 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:12.261234 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:07:12.270195 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:12.315534 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:12.316032 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:12.345454 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:12.378675 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:12.407748 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:12.415745 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:07:12.662607 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:12.719259 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:07:12.733860 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:12.749019 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:12.752560 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:12.796826 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:12.800097 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:12.815819 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:12.821519 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:07:12.846388 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:12.870375 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.005522 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:13.007701 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:13.007740 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:13.008708 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:13.012382 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:13.019698 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:13.042080 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:13.043066 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.044078 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:13.120251 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:13.166659 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:13.251600 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:13.276885 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:13.291330 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:13.313428 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:13.350395 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.363564 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.367336 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:13.395653 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:13.395929 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:13.415066 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:13.453722 7487 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:07:13.453913 7487 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:07:13.454676 7487 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:07:13.455507 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:07:13.463441 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:07:13.463765 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:13.463874 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:07:13.464308 7487 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:07:13.483083 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:13.506158 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:13.511279 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.533226 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.562955 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.583935 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:13.684458 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:13.708464 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:13.758962 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:13.774693 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:13.783274 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:13.791888 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.815550 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:13.833441 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:13.855258 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:13.977859 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:14.100373 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:14.108650 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:14.117532 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:14.118288 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:14.125538 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:14.131397 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:14.139281 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:07:14.156359 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:14.162897 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:14.242058 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:14.367925 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:14.402444 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:14.407347 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:14.408955 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:14.414294 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:14.421820 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:14.436125 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:14.514119 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:14.575408 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:07:14.576422 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:14.613844 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:14.657928 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:14.742602 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:14.763085 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:14.782860 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:14.797493 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:14.824648 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:14.828472 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:14.832154 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:14.852342 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:14.902655 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:14.915127 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:14.918054 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:14.945763 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:14.973636 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:14.994293 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:15.005897 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:15.016485 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:15.231665 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:15.259490 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:15.261286 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.276356 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:15.285517 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.288739 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.307788 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:15.379771 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:15.531859 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:15.541952 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:15.543125 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:15.543289 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:07:15.551829 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:15.564237 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.570137 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:15.574378 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.581784 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:15.627351 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:15.666133 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:15.759631 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:15.824604 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:15.829372 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:15.833420 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:15.850092 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:15.850769 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.855185 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:15.865943 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.871273 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:15.940687 7279 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:07:15.940894 7279 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:07:15.941612 7279 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:07:15.945408 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:07:15.946077 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:07:15.953076 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:07:15.953214 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:07:15.953460 7279 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:16.117307 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:16.138466 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:16.155478 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:16.156142 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.162168 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.174381 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:16.189450 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:16.213881 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:16.222424 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:16.306167 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:16.308825 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:16.324067 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:16.327358 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:16.345469 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.373380 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:16.373646 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.424985 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:16.495150 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:07:16.608045 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:16.686764 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:07:16.699787 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:16.717680 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:16.727607 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:16.761512 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:16.765411 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:16.768119 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.776015 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.788642 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.818006 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:16.902946 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:16.922609 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:16.932400 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.947643 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.949153 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:16.950870 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:16.954079 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:17.151687 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:17.203510 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:17.227025 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:17.236737 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:17.305256 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:17.305716 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:17.319394 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:17.327785 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:17.398240 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:17.399106 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:17.399246 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:17.423656 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:07:17.425405 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:17.545325 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:17.556778 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:17.585565 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:17.613278 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:17.644343 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:17.655535 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:17.661217 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:17.670876 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:17.674860 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:17.688277 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:17.742741 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:07:17.851751 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:17.858448 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:17.866938 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:17.867863 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:17.907455 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:17.923035 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:17.971311 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:07:18.043466 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:07:18.057030 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:18.134819 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:18.150913 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:18.156903 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:18.158443 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:18.176955 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:18.178372 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:18.189595 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:18.248431 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:18.298159 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:18.364689 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:18.371062 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:18.390851 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:18.400419 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:18.417896 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:18.427569 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:18.456962 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:18.464087 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:18.475167 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:18.489468 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:18.562815 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:18.627141 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:18.648481 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:07:18.715934 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:18.727772 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:18.738536 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:18.742373 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:18.764549 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:18.766453 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:18.799355 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:18.844414 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:18.913162 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:18.937454 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:19.032706 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:19.036195 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:19.065652 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.107995 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.112486 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:19.114914 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:07:19.116932 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:19.233489 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:19.275956 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:19.294263 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:19.350890 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:19.361088 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.368152 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.375787 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:19.381542 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:19.482195 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:19.492013 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:19.516376 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:19.608454 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:19.609416 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:19.636241 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.639461 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.667022 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:19.667359 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:19.672240 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:19.676254 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:07:19.794517 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:07:19.811096 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:19.816026 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:19.821627 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.853957 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:19.859766 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:19.894055 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:19.895655 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:19.952080 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:20.117758 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:20.129464 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:20.149531 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:20.165692 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:20.177502 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:20.182125 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:20.198876 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:20.203663 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:20.213575 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:20.276850 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:20.417927 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:20.470005 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:20.480370 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:20.484930 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:20.523535 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:20.532299 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:20.533232 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:20.582222 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:20.583107 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:20.634864 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:20.647137 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:20.712524 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:20.726600 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:20.732417 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:20.733794 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:20.734174 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:20.745183 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:20.745976 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:20.965482 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:20.982353 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:21.031162 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:21.038497 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:21.040243 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:21.047338 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:21.071069 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:21.087055 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:21.099270 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:21.100009 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:21.224447 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:21.248456 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:21.261583 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:21.270901 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:21.313709 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:21.314332 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:21.341262 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:21.354264 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:21.387913 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:21.455845 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:21.469281 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:21.615765 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:07:21.621426 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:21.625207 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:21.640741 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:21.645061 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:21.656648 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:21.679324 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:21.838841 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:07:21.910703 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:21.967460 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:21.970494 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:21.982107 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:22.001103 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:22.043530 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:22.114549 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:22.115895 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:22.136055 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.139485 28725 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:22.139770 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:22.183873 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.205184 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:22.215659 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:22.224571 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.243175 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.266440 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:22.269990 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.291944 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:22.403417 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:22.438694 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:07:22.516590 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:22.561730 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.565723 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:22.576356 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:22.576402 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.611512 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:22.619477 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:22.693445 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:22.773568 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:22.830526 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:22.863375 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:22.866526 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:22.867210 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:22.877918 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:22.886371 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:22.888520 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:22.907565 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:22.912144 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:22.914839 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:07:22.991021 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:23.061165 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:23.062614 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:07:23.093320 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:23.129042 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:23.147493 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:23.150674 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:23.155772 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:23.156636 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:23.161839 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:23.166725 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:23.231361 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:07:23.295291 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:23.456329 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:23.460987 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:23.472648 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:23.474674 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:23.483379 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:23.512368 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:23.553093 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:23.573976 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:23.576148 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:23.671334 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:07:23.707532 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:23.711575 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:23.726070 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:23.745512 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:23.757565 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:23.767446 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:23.883122 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:23.988756 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:24.001277 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:24.013470 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:24.019868 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:24.027025 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:24.102283 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:24.102948 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:07:24.115850 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:24.127516 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:24.149793 28941 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 64285us
I20240501 14:07:24.158895 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:24.198854 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:24.251514 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:24.282943 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:24.298128 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:24.330585 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:24.375455 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:24.382584 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:24.398451 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:24.400738 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:24.503960 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:24.515367 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:24.553365 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:24.682646 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:24.688467 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:24.689218 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:24.695305 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:24.699091 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:24.718803 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:24.824785 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:24.854964 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:25.017493 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:25.023015 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:25.050109 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:07:25.053869 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:25.063256 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:25.064078 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:25.090234 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:25.107618 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:25.112792 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:25.137981 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:25.155036 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:25.269040 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:25.304693 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:25.307708 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:25.350875 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:25.383553 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:25.403941 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:25.409003 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:25.480264 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:25.530866 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:25.601699 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:25.642164 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:25.645395 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:25.654040 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:25.712472 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:25.740140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:25.749805 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:07:25.754814 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:25.758042 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:25.770516 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:25.771173 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:25.775415 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:25.839496 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:25.841653 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:25.855713 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:25.915545 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:25.944875 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:25.970983 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:26.017529 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:26.121615 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:26.315405 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:26.342105 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:26.376538 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:26.522552 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:26.545780 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:26.568486 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:26.584110 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:26.584587 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:26.592314 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:26.609633 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:26.640395 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:07:26.640772 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:26.651918 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:26.665067 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:26.674827 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:26.688308 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:26.713878 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:26.724584 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:26.726842 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:26.735507 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:26.865978 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:27.004535 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:27.007149 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:27.011296 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:27.016997 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:27.030697 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:27.037827 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:27.038720 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:27.108544 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:27.141677 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:27.246798 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:27.281172 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:27.297758 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:27.306272 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:07:27.321908 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:27.350883 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:27.364884 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:27.434039 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:27.434547 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:27.434803 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:27.436518 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:27.438606 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:07:27.449558 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 14:07:27.502446 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:27.523437 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:27.566691 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:27.568663 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:27.593405 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:27.615309 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:27.623422 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:27.716634 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:27.991492 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:07:27.992686 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:28.030381 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:28.044193 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:28.080035 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:07:28.083175 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:28.096088 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.107764 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.113705 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.117596 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:28.145649 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:28.167531 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:28.181088 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:28.303400 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:28.303848 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:28.306936 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:28.321789 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:28.334663 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.343858 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.362156 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:28.483893 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:28.510658 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:28.533115 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:28.584270 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:28.585659 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:28.678493 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:07:28.697919 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:28.701020 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:28.702986 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:07:28.718499 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.727620 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.735788 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:28.736471 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:28.743080 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:28.781893 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 60711.5us
I20240501 14:07:28.783813 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:28.789402 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:28.827106 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:28.872608 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:28.889372 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.893397 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:28.919473 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:29.042569 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:29.139072 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:29.155552 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:29.156225 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:29.178853 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:29.182869 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:29.186338 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:29.199203 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:29.212750 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:29.294021 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:29.336228 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:29.456259 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:29.457389 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:29.473588 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:29.478924 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:29.492226 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:29.519101 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:29.540787 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:29.639161 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:29.660353 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:29.722426 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:29.732487 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:29.732586 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:29.736909 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:29.743674 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:29.758932 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:29.758916 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:29.766657 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:29.784723 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:29.789121 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:30.018352 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:30.019245 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:30.064826 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:30.083164 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:30.090394 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:30.113983 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:30.121568 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:30.151608 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:30.154978 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:07:30.317065 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:30.335610 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:30.339790 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:30.340296 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:30.340584 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:30.349324 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:30.483914 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:30.533028 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:30.556552 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:30.581249 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:30.597513 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:30.731829 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:30.736835 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:30.744760 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:30.762141 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:30.762841 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:30.778898 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:30.800443 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:30.802793 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:30.804220 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:30.830019 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:30.851969 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:07:30.885650 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:30.889243 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:30.893296 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:30.917708 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:30.919526 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:31.010124 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:31.095738 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:31.186731 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:07:31.214167 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:31.243903 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:31.321660 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:31.374739 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:31.398988 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:31.462019 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:31.476684 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:31.477767 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:31.489830 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:31.513002 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:31.522161 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:31.566128 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:31.567831 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:31.585211 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:31.613186 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:31.620332 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:31.658501 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:31.837026 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:31.861246 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:31.879559 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:31.880517 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:31.881539 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:31.890393 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:31.898237 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:31.922494 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:31.956920 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:32.011215 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:32.035398 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:32.149519 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:32.184424 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:32.196684 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:32.222209 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:32.225319 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:32.233458 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:32.234344 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:32.252811 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:32.267079 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:32.280239 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:32.319482 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:32.341918 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:32.357697 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:32.389606 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:32.390705 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:32.392702 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:32.429819 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:32.466895 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:07:32.594199 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:32.763236 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:07:32.795717 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:32.928463 10318 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:07:32.928795 10318 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:07:32.929929 10318 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
I20240501 14:07:32.938222 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:32.939929 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:32.940411 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:07:32.940567 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:07:32.941717 10490 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:07:32.958523 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:32.961704 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.005264 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:33.014429 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:07:33.025949 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.043860 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:33.061743 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:33.090711 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 14:07:33.106798 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:33.113273 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:33.113737 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.126891 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:33.160413 30730 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 59815.3us
W20240501 14:07:33.183857 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:33.196836 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:33.198369 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:33.204128 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.206701 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:33.274288 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:33.317328 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:33.378695 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:33.390292 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:33.445765 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:07:33.498111 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:07:33.499413 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:33.501235 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:33.527154 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.531199 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:33.541972 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:33.545351 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:33.566565 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:33.608630 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:33.646078 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:33.819619 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:33.827629 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 14:07:33.854689 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.858904 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:33.860324 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:33.895782 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.899353 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.901212 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:33.906579 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:33.908008 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:33.953923 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:33.963251 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:33.965557 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:33.991107 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:33.997817 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:34.013607 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:34.073849 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:34.499059 10219 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:07:34.499276 10219 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:07:34.500115 10219 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:07:34.505906 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:34.509690 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:34.544463 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:34.550202 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:34.555927 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:34.556027 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:07:34.584769 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 64265.9us
W20240501 14:07:34.586097 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:07:34.587977 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:07:34.588244 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:07:34.588719 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:34.588757 10219 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:07:34.591694 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:34.640331 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:34.646775 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:34.667801 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:34.691325 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:34.727942 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:34.736503 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:34.755299 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:34.764766 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:34.779508 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:34.791702 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:34.798877 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:34.837541 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:34.865111 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:34.929903 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:34.989560 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:35.021977 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:35.031917 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:35.032176 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:35.035470 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:35.083117 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:35.083079 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:35.167044 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:35.227365 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:35.233112 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:07:35.256937 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:35.323858 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:35.402575 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:35.432085 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:35.475914 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:35.490940 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:35.505571 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:35.643924 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:35.690006 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:35.693732 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:35.696079 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:35.713907 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:35.714778 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:35.728322 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:35.729924 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:35.788820 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:35.872288 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:35.958690 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:35.989825 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:36.015347 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:36.017009 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:36.029430 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.031473 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:36.037251 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:36.044642 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:36.069377 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:36.118932 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:36.147013 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:36.210853 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:36.233793 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:36.259743 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:07:36.262367 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:36.276468 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:36.292472 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:36.308776 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:07:36.309310 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.310714 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.313372 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.315363 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:36.622454 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:36.637725 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:36.652652 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:36.658708 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:36.687804 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:36.688019 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.731065 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.751026 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:36.779433 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:36.816109 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:36.841794 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:36.848826 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.869589 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:36.900976 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:36.925199 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:37.057215 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:37.128890 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.132548 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:37.147338 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:37.165064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.166958 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:37.168112 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.203868 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:37.316113 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:37.323061 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:37.323192 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:37.323439 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:37.334846 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:37.366506 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:37.404465 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:37.410240 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:37.424546 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:37.428575 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.442023 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.459786 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.465488 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:37.500227 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:07:37.662122 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:37.703861 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:37.713827 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.729341 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:37.730501 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:37.747676 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:37.762344 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:37.789439 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:37.814152 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:38.015496 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:07:38.016415 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:38.041123 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:38.051417 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:38.078977 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:38.094588 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:38.110682 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:38.127539 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:38.140771 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:38.279778 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:38.294582 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:38.300156 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:07:38.303310 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:38.322253 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:38.332990 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:38.342012 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:07:38.375496 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:38.377436 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:38.380201 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:38.386376 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:38.402281 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:38.441171 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:38.598397 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:38.630851 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:07:38.663432 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:38.714293 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:38.718858 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:38.723995 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:38.733103 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:38.760288 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:38.783820 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:38.783914 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:38.811432 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:38.822773 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:38.880585 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:38.882555 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:38.883041 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:07:38.885726 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:38.907582 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:38.926403 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:38.939509 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:07:38.941850 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:39.240453 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:39.347769 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:39.426604 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:39.488399 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:39.497869 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:39.499120 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:39.536850 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:39.542356 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:39.547056 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:39.592643 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:39.599818 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:39.605409 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:07:39.609979 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:39.632401 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:39.684736 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:39.685006 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:39.686100 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:39.766103 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:39.827404 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:39.834882 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:39.920347 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:39.920585 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:39.929301 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:39.934005 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:39.958461 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:39.960925 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:39.970266 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:39.984650 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:40.017923 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:40.061578 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:40.259337 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:40.266726 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:40.280107 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:40.319659 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:40.325524 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:40.353549 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:40.357978 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:40.370064 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:40.381362 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:40.390079 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:40.424822 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:40.462832 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:40.482774 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:07:40.484088 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:40.494264 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:40.595875 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:40.686892 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:40.795303 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:40.885318 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:40.918690 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:40.929738 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:40.978735 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:40.984658 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:40.996474 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:41.024988 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:41.037282 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:41.051465 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:41.066898 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:41.071046 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:41.122092 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:41.127210 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:41.195930 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:41.332388 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:41.338006 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:41.339524 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:41.349663 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:41.376492 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:41.397250 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:41.403535 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:41.537740 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:07:41.540831 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:41.590557 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:41.618661 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:41.625967 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:41.635731 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:41.642745 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:41.665025 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:41.670941 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:41.777205 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:41.962563 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:41.968829 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:41.982769 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:41.989645 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:41.989821 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:42.022557 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.046942 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:42.053277 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:42.064201 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.070315 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.078871 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:42.089251 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:42.099674 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.129068 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.136013 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:42.148931 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:42.164021 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:42.334194 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:42.392939 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:42.493119 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:42.534471 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:07:42.565779 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:42.578097 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:42.680325 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:42.687639 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:42.712865 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:42.736083 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:42.742938 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:42.760600 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:42.777609 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.831755 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.836042 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.855438 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:42.863727 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.907343 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:42.920389 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:42.943063 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:42.950647 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:43.041216 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:43.048661 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:43.087981 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:43.211484 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:43.224342 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:43.227711 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:43.234076 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:43.246474 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:07:43.247287 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:43.271667 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:43.351043 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:07:43.397256 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:07:43.399955 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:43.399899 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:43.541883 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:43.549858 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:43.554984 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:43.563138 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:43.564695 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:43.571178 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:43.577289 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:43.591307 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:43.593358 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:43.625805 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:43.682168 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:07:43.689009 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:43.785423 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:43.798533 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:43.822971 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:43.823657 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:43.826359 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:43.870741 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:43.998267 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:07:44.063982 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:44.072288 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:44.073189 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:44.081755 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:07:44.094249 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.094424 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.095256 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:44.097484 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:44.117496 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:44.167099 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:44.287997 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:44.356989 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:44.359344 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:44.361696 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.380502 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.381551 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:44.420322 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:44.446959 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:44.460395 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:44.500182 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:44.563992 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:44.657037 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:44.692245 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.706053 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:44.720865 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.725023 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:44.744256 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:44.800786 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:44.847373 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:44.880882 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:44.884212 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:44.884979 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:44.899411 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:44.903786 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.905459 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:44.921216 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:45.013242 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:45.071244 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:07:45.145656 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:45.181739 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:45.182183 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:45.205369 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:45.215320 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:45.219079 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:45.235993 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:45.274730 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:45.330955 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:45.372030 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:45.394915 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:45.454982 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:45.462013 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:45.462014 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:45.462306 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:45.474746 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:45.485321 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:45.493500 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:45.518975 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:07:45.668879 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:45.752466 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:45.760143 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:45.762207 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:45.778385 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:45.790804 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:45.794589 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:45.815233 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:45.964697 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:45.969873 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:46.005415 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:46.066427 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:46.072942 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:46.078012 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:46.079555 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:46.100059 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:46.103520 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:46.105348 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:46.119622 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:46.247306 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:46.260788 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:46.265920 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:46.283403 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:46.292851 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:46.308442 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:46.340086 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:46.379585 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:46.592991 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:46.620954 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:46.656158 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:46.678210 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:46.690582 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:46.698137 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:46.702250 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:46.703935 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:46.711431 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:46.715812 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:07:46.848857 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:46.851727 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:46.854499 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:46.900578 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:46.912050 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:46.930925 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:46.952344 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:46.997452 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:47.041450 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:47.143005 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:47.154690 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:47.156797 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:47.412958 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:47.420397 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:47.428145 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:47.480154 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:47.517529 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:47.537997 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:47.549069 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 14:07:47.550892 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:47.554786 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:47.588532 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:47.595235 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:47.599222 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:47.617972 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:47.624625 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:47.648613 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:47.655215 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:47.700809 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:47.701433 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:47.720990 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:47.730791 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:47.926343 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:47.928409 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:47.934058 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:47.941398 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:47.952461 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:47.959187 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:48.002558 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:48.137118 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:48.185019 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:48.187520 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:48.195259 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:48.229911 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:48.236018 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:48.247668 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:48.248221 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:48.251785 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:48.298681 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:48.333909 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:07:48.378432 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:07:48.383376 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:48.466799 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:48.477715 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:48.497381 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:48.510895 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:48.537444 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:48.544251 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:48.567075 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:07:48.701179 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:48.701745 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:48.717846 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:48.782488 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:48.788201 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:48.794790 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:48.799077 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:48.815392 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:48.832129 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:48.833042 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:48.833885 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:48.914336 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:07:48.982546 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:49.043198 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:07:49.057235 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:49.082509 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:07:49.119673 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:49.130054 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.137140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:49.155654 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.160243 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.164176 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:49.337697 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:49.363502 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:49.454079 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:49.479668 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:49.507344 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.508608 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.520032 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.563364 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.577761 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:49.620585 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:49.634555 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:49.634773 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:49.665962 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:49.686578 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:49.708714 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:49.711319 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:49.713569 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.734402 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:49.850665 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:49.850919 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:49.940532 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:49.951788 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.957237 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.963094 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:49.979251 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:49.988282 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:50.024977 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:50.069100 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:50.070662 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:50.185442 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:07:50.192493 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:50.195281 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:50.215241 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:50.218166 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:50.222851 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:50.247721 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:50.377190 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:50.431842 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:50.445729 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:50.467396 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:50.494364 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:50.498361 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:50.515668 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:50.526966 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:50.536718 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:50.543645 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:50.547148 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:50.649449 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:07:50.766156 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:07:50.781656 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:50.844218 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:50.852768 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:50.893127 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:50.896080 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:50.919858 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:07:50.925346 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:50.928006 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:50.938848 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:50.981909 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:50.988503 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:51.001222 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:07:51.014052 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:51.027218 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.051214 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.058396 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.059229 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:51.207649 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:51.215739 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:51.281859 12522 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:07:51.282099 12522 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:07:51.282979 12522 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:07:51.288575 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:51.302242 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:07:51.302430 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:07:51.302752 12522 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:51.313606 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:51.314417 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:51.316587 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.343840 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.352902 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:51.374742 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:51.452356 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:51.475666 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:07:51.494297 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:51.579936 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:51.636891 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:51.642510 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:51.652195 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.684657 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:51.709466 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.722566 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:51.879248 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:51.914947 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:51.917239 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:51.931978 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:51.933089 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.936434 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:51.956728 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:51.957976 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:52.020254 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:52.078706 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:52.189945 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:07:52.229183 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:52.236512 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:52.263507 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:52.303637 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:07:52.316349 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:52.329486 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:52.342166 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:52.360114 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:52.368922 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:52.376929 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:52.383914 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:52.384008 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:52.435182 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:52.488914 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:52.497241 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:52.506006 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:52.526950 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:52.661090 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:07:52.719487 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:52.755122 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:52.769371 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:52.800652 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:52.813364 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:52.817873 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:52.823896 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:07:52.835497 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:52.844512 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:52.846091 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:52.902593 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:52.978144 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:53.032497 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:53.033071 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:53.035413 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.045486 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:53.051645 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.058612 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:53.078444 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:53.105396 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:07:53.305378 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.306839 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:53.312693 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.326156 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:53.326267 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:53.335394 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:53.335383 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:53.343133 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:53.357620 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:07:53.425705 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:53.425887 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:07:53.465170 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:07:53.620532 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:53.620888 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:07:53.650102 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:53.653174 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.672983 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.707089 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:53.727988 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:53.740496 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.742985 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:53.799562 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:53.829185 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:53.832479 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:53.842782 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:53.849289 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:53.874819 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:53.894098 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:53.903887 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:54.061604 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 14:07:54.111284 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:54.115478 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:54.129050 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:54.137511 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:54.139955 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:54.142904 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:54.268430 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:54.284920 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:07:54.289472 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:54.310812 13337 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:07:54.311410 13337 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:07:54.312263 13337 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:07:54.313586 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:07:54.314220 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:07:54.314741 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:07:54.314924 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:07:54.315663 13337 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:54.317677 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:07:54.335505 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:54.406229 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:54.427206 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:54.428722 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:54.428799 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:54.439620 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:54.483639 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:54.535187 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:54.666618 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:54.714084 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:54.715804 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:54.717813 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:54.736991 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:07:54.742940 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:54.743312 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:54.759476 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:54.763288 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:54.770404 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:54.776676 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:54.919921 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:07:54.980224 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:07:55.053422 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:55.064980 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:55.068097 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:55.083132 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:55.117033 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:55.141083 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:55.141891 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:55.162961 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:55.206724 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:55.240283 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:55.265986 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:07:55.278242 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:55.321868 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:55.349159 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:55.353436 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:55.356446 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:55.386858 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:07:55.600976 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:55.617933 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:55.622608 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:55.644780 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:55.650090 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:55.665163 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:55.675825 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:55.708536 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:07:55.722316 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:55.726382 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:07:55.727304 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:55.756284 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:55.790920 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:07:55.834313 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:07:55.840874 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:55.850193 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:55.854970 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:55.856855 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:55.863554 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:55.879496 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:55.913977 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 64899.2us
W20240501 14:07:56.011165 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:07:56.094837 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:07:56.130942 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:56.169661 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:56.169934 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.192515 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.193012 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.197319 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:56.321590 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:56.403549 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:56.478758 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:56.508487 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:56.533818 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:56.538478 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.562016 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.574080 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.575397 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:56.777405 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:56.781473 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.785317 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:56.797760 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:56.829180 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:56.844218 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:56.857686 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:56.860709 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:56.871297 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:56.884883 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:07:56.986644 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:07:57.097555 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:57.112107 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:57.115770 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:57.118222 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:57.118219 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:57.176649 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:57.279670 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:07:57.326429 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:57.335439 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:57.335753 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:07:57.352785 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:57.355190 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:57.374554 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:57.395383 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:57.419857 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:57.455581 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:57.485033 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:57.486917 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:07:57.649645 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:57.659555 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:57.664986 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:07:57.676654 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:57.690508 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:57.698295 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:57.701133 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:57.726368 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:57.785621 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:57.814788 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:07:57.816067 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:07:57.955368 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:07:57.965307 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:07:57.971838 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:07:57.997567 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:58.023181 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:58.031343 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:58.033252 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:07:58.037377 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:58.044399 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.062767 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.089131 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.135823 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:07:58.149585 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:07:58.151010 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:58.166700 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.172436 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:58.193112 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:58.198726 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:07:58.198899 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:58.230561 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:58.332305 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:07:58.415766 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:07:58.500429 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:58.524892 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:58.533900 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.539281 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.540179 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:58.562193 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:07:58.565735 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:58.590824 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:07:58.655125 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:07:58.860710 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:58.862926 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.865300 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:58.873606 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:58.875339 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:58.878365 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:58.898123 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:59.039242 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:07:59.076700 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:59.100555 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:59.127554 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:59.129323 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:59.151262 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:59.152997 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:59.161881 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:59.175509 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:59.212586 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:07:59.293314 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:59.366283 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:07:59.461560 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:07:59.472698 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:59.506898 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:59.509192 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:59.510874 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:59.529860 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:07:59.543927 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:07:59.586839 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:07:59.590492 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:07:59.636552 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:07:59.697420 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:07:59.709633 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:07:59.710479 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:59.719754 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:59.732998 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:07:59.738862 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:07:59.775269 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:07:59.918399 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:07:59.990295 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:00.006662 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:00.009387 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:00.011597 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:00.036671 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:00.072767 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:00.097275 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:00.097431 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:00.103091 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:00.119676 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:00.216254 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:08:00.221923 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:00.229864 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:00.240442 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:00.243616 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:00.249943 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:08:00.276588 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:00.278138 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:00.444305 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:00.486120 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:00.506577 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:00.518796 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:08:00.564056 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:00.576879 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:08:00.577378 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:08:00.584726 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:00.604355 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:00.648352 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:00.666234 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:00.703449 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:00.704784 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:00.715412 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:08:00.736467 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:00.844895 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:08:00.877522 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:00.930474 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:00.961987 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:00.979135 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:00.995689 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:01.013005 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:01.063133 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:08:01.071185 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:01.099992 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:01.225044 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:01.230600 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:01.250895 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:01.265688 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:01.273465 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:01.294569 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:01.476037 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:01.506170 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:01.544642 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:08:01.560709 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:01.571980 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:01.592128 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:08:01.615027 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:01.620846 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:08:01.631855 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:01.633597 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:01.636714 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:01.663105 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:01.683140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:01.742148 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:01.781489 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:08:01.788010 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:01.820633 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:01.838968 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:01.840799 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:08:02.010521 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:02.148612 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:02.153959 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:02.171365 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:02.172678 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.176262 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.179874 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.198968 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:02.373899 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:02.399780 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:02.424269 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:02.457872 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:02.459131 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:02.475183 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.506362 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:02.507547 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:02.517803 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.550058 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:02.601998 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:02.624764 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:02.705535 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:08:02.733263 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.739169 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:02.745396 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.748122 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:02.760649 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:02.771453 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:02.835387 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:02.844867 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:08:02.868932 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:02.949759 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:08:02.982960 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:03.031701 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:08:03.077054 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:03.081216 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:03.088958 13752 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:08:03.089177 13752 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:03.089991 13752 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:08:03.090761 30794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:08:03.092468 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
W20240501 14:08:03.092952 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:03.093091 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:03.093355 13752 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:03.108304 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:03.117942 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:03.118794 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:03.126485 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:03.164155 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:03.171504 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:03.181120 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:08:03.190917 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 57707.6us
I20240501 14:08:03.193522 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:03.195798 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:08:03.217834 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:03.277982 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:03.296408 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:08:03.309520 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:03.315261 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:03.330569 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:03.334576 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:03.500560 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:08:03.616729 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:03.616788 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:03.671942 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:08:03.673234 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:03.815315 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:03.823185 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:03.890859 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:03.891705 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:03.901114 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:03.910480 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:03.937958 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:03.966070 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:04.006551 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:04.006978 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:08:04.021158 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:04.039961 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:04.045706 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:04.055244 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:04.112744 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:08:04.125293 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:04.216162 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:08:04.377485 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:08:04.383626 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:04.399425 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:04.400799 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:04.416747 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:04.420075 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:04.421638 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:04.450976 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:04.485826 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:08:04.687980 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:04.687985 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:04.781340 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:04.794133 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:04.802933 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:04.869889 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:04.883157 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:04.897532 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:04.912741 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:04.920468 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:04.932119 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:04.947618 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:08:04.963824 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:08:04.969261 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:04.997977 28680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 81962.9us
W20240501 14:08:05.003796 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:05.047130 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:05.052827 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:05.061959 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:05.068080 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:05.124998 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:05.204293 14450 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:08:05.204495 14450 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:08:05.205190 14450 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:08:05.206854 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:05.207334 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:05.207438 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:08:05.207669 14450 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:05.355780 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:05.363658 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:05.389514 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:05.434967 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:05.473374 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:05.480181 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:05.525332 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:05.526207 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:05.534955 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:05.553436 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:05.572412 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:05.581208 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:08:05.617564 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:05.650342 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:05.683445 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:05.773136 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:05.781227 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:05.782539 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:05.787647 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:05.802120 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:05.805514 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:05.817492 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:05.838470 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:05.848671 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:05.911756 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:08:05.967365 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:06.077863 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:06.098997 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:06.109073 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.112692 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:06.112811 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:06.141471 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.182072 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.305526 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:06.338137 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:06.339752 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:06.345651 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:06.367655 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:06.375536 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:06.379565 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.417102 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:06.418889 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:06.424955 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:06.460847 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:06.580857 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:06.602780 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:06.607466 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.615568 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:06.621543 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.626909 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:06.821559 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:06.906481 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:06.910797 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:06.925611 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:08:06.933266 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.951059 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:06.955297 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:06.968755 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:06.997644 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:08:07.046994 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:07.184986 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:07.195554 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:07.210072 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:07.213029 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:07.246814 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:07.249650 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:07.254351 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:07.259944 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:07.288381 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:07.346199 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:07.503096 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:07.527662 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:07.541049 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:08:07.603336 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:07.619158 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:07.638062 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:07.654372 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:07.672235 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:07.675393 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:07.683072 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:07.686167 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:07.707175 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:07.709825 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:07.732553 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:07.761260 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:07.768741 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:07.771276 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:07.786810 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:07.792534 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:07.872706 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:07.886723 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:07.903368 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:08:08.024276 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:08.034505 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:08:08.035372 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:08.043143 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:08.053064 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:08.059763 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:08.075412 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:08.156188 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:08.218796 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:08.268311 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:08.272748 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:08.295017 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:08.305666 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:08.322767 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:08.352798 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:08.354739 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:08.359030 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:08.554680 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:08:08.582700 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:08.593953 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:08.594179 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:08.616102 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:08.619676 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:08.642133 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:08.643760 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:08.710157 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:08:08.738571 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:08.778416 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:08.779007 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:08.876034 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:08.890851 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:08.948580 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:08.951267 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:08.976361 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:08.976675 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:08.979259 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:09.010936 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:09.148181 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:09.150609 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:09.169574 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:09.177871 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:09.185096 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:09.191721 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:09.200260 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:09.241020 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:08:09.303615 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:08:09.354591 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:09.426250 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:09.548758 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:09.565595 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:09.581796 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:09.585482 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:09.594604 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:09.622244 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:09.624210 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:09.625272 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:09.630839 13752 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:08:09.631064 13752 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:09.631951 13752 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:08:09.634953 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b9db597e654ed48b679abf58b8a2c8" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8fc5d8222c164a679e7a99966da1eaaf" is_pre_election: true
W20240501 14:08:09.636309 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:09.636610 30862 leader_election.cc:341] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Illegal state: cannot vote while shut down
I20240501 14:08:09.636734 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:09.637374 13752 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:09.638878 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:09.648451 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:09.672778 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:09.725080 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:09.732088 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:09.739830 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:09.768237 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:09.772950 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:09.777184 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:10.018620 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:10.086346 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:10.109866 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:10.208739 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:10.240352 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:08:10.242074 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:08:10.273129 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:10.303479 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:10.313738 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:10.330209 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:10.335204 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:10.352737 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:10.373541 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:10.406302 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:10.414228 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:10.431325 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:10.456269 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:08:10.458331 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:10.460150 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:10.520552 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:10.521221 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:10.523924 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:10.528637 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:10.593027 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:10.637080 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:08:10.718573 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:10.757371 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:10.765631 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:10.771414 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:10.807368 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:10.818955 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:10.835304 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:10.963315 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:08:10.989926 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:11.054775 30729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:08:11.092623 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:11.102135 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:11.124698 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:11.140430 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:11.141875 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:11.159230 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:11.166584 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:11.186758 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:11.193415 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:11.213675 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:11.223116 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:11.277665 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:11.281567 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:11.296350 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:11.327105 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:11.381429 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:11.514992 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:11.611549 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:11.613765 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:11.726908 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:11.733168 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:11.744709 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:11.766734 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:11.778364 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:11.798890 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:08:11.807960 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:11.909649 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:11.992408 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:12.019637 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:12.029626 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:12.036149 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:12.049353 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:12.054636 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:12.060631 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:12.085237 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:12.118994 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:12.134806 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:12.237280 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:12.243592 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:12.249913 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:12.259549 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:12.266824 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:12.282666 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:12.351040 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:12.581869 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:12.593482 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:12.616709 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:08:12.621714 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:12.622983 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:12.628649 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:12.629703 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:12.631125 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:12.645292 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:12.656780 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:12.725875 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:12.787508 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:12.805948 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:12.822587 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:12.823001 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:12.844539 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:12.846091 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:12.919296 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:12.950145 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:08:12.990742 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:08:13.045053 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:08:13.069746 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 14:08:13.118796 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:13.144351 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:13.144392 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.160501 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.168212 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:13.196584 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:13.219815 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:13.260934 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:13.287087 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:13.398957 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:13.403860 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:13.427358 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:13.442191 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.448619 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.455178 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:13.480192 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:13.586308 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:08:13.613236 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:08:13.646507 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:08:13.722775 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240501 14:08:13.724277 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:13.755548 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:13.769485 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:13.779543 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:13.784212 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.811991 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.855391 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.858042 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:13.868779 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:13.873831 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:13.913460 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:13.913880 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:13.952025 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:13.994246 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:14.004165 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:14.014503 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:14.085839 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:14.147164 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:08:14.150496 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:14.232123 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:14.233821 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:14.248297 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:14.260165 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:14.292034 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:14.310839 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:14.315725 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:14.316394 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:14.341305 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:14.423758 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:14.492601 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:14.536860 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:14.539960 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:14.543340 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:14.547704 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:14.554847 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:14.566241 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:14.579589 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:08:14.618398 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:14.760608 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:14.821676 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:14.827440 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:14.848580 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:14.865772 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:14.891328 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:14.891522 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:14.986552 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:08:15.069231 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:15.075727 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:08:15.090416 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:15.137436 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:15.167779 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:15.169997 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.178532 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:15.225399 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.243292 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:15.256420 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:15.316928 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:08:15.355371 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:15.413983 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:15.433853 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.437511 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.439046 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:15.456460 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:15.483530 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:15.533582 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:08:15.649672 28810 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:08:15.656127 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:15.696081 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:15.703253 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:08:15.713014 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:15.728834 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:15.729331 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.737969 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.755558 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:15.785619 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:15.888190 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:15.922466 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:15.927464 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.950492 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:15.975570 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:15.981664 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:15.982010 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:15.996697 28941 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:08:16.097438 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:08:16.170358 30730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:08:16.182739 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:16.184967 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:16.198508 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:16.204262 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:16.210986 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:16.213968 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:16.244460 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:16.330107 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:08:16.362916 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:08:16.383510 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:08:16.398710 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:16.507155 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:16.541615 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:16.561060 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:16.564381 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:16.574352 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:16.594172 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:16.598130 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:16.603780 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:16.655807 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:08:16.688024 30729 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:08:16.701226 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:08:16.726759 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:16.743240 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:16.753542 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:16.762995 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:16.781004 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:16.798851 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:16.803298 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:16.829809 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:17.010358 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:17.019373 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:17.022248 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:17.026127 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:17.052196 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:17.066844 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:17.117334 30862 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:08:17.195151 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:17.266369 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:17.276490 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:17.300494 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:17.301755 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:17.307000 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:17.326926 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:17.328814 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:17.563326 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:17.575770 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:17.578933 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:17.584566 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:17.605432 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:17.624851 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:17.625165 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:17.690008 28941 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:08:17.709134 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:17.710690 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:08:17.733490 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:17.874060 16158 raft_consensus.cc:998] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.137s)
I20240501 14:08:17.875243 16158 consensus_queue.cc:237] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:17.909088 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:17.911310 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:17.912585 30793 raft_consensus.cc:1261] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:08:17.913542 16525 consensus_queue.cc:1035] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240501 14:08:17.917366 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:17.917296 16158 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:17.926642 28595 catalog_manager.cc:5484] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 8 to 9, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:17.929028 30793 raft_consensus.cc:2932] T f4402de0b48041c383568feb9365147e P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:17.936928 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:17.995877 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:08:18.000571 16551 raft_consensus.cc:998] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.203s)
I20240501 14:08:18.001518 16551 consensus_queue.cc:237] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
W20240501 14:08:18.033681 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:18.034145 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:18.034511 28745 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:18.035288 16545 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:18.039647 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:18.039682 16325 raft_consensus.cc:998] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.315s)
I20240501 14:08:18.040555 16325 consensus_queue.cc:237] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:18.041635 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:08:18.051465 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:18.053759 16545 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:18.054224 28745 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:18.061430 28599 catalog_manager.cc:5484] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
W20240501 14:08:18.076388 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f4402de0b48041c383568feb9365147e: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:18.077343 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:18.078068 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:18.083307 28745 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:18.084115 16325 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:18.119856 16484 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:18.123014 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:18.126510 28744 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:18.130514 16546 raft_consensus.cc:998] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.253s)
I20240501 14:08:18.131457 16546 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:08:18.132532 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 14:08:18.134194 28595 catalog_manager.cc:5484] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } }
I20240501 14:08:18.140766 13752 raft_consensus.cc:998] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.383s)
I20240501 14:08:18.141607 13752 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:18.168725 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:18.186345 30793 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:08:18.187160 16272 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:08:18.222167 16546 raft_consensus.cc:998] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.079s)
I20240501 14:08:18.223184 16546 consensus_queue.cc:237] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:18.235144 29136 raft_consensus.cc:1261] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:08:18.237097 16551 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:08:18.237859 16551 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.240506 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:18.242218 30794 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.245348 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:18.253630 28595 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 7 to 8, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.257932 29006 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer a0e901103604414c829b40edf9f24c28: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:08:18.258734 13752 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:08:18.260237 16272 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.265118 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:18.265744 29136 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.269905 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:18.273440 16564 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:08:18.281092 29007 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
W20240501 14:08:18.288090 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:18.292174 28595 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 10 to 11, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 2 leader_uuid: "a0e901103604414c829b40edf9f24c28" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:08:18.313797 28595 catalog_manager.cc:5484] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.343897 16250 raft_consensus.cc:998] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.413s)
I20240501 14:08:18.344976 16250 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:18.359179 16583 raft_consensus.cc:998] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.172s)
I20240501 14:08:18.361356 16583 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:18.362979 29007 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:18.363406 29006 consensus_queue.cc:237] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
W20240501 14:08:18.366400 28940 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Couldn't send request to peer 3a3972b170c54344a4ba61f176f837ba. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:18.372015 30793 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:08:18.373004 16272 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:08:18.377938 30926 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:18.387501 16272 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.389212 30793 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.392576 16565 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:08:18.392791 16565 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:08:18.393616 16565 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
I20240501 14:08:18.395238 29006 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer a0e901103604414c829b40edf9f24c28: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 14:08:18.395259 28940 consensus_peers.cc:485] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Couldn't send request to peer cd3c04ebb6c84b15a1b3adc748737094. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:18.395900 16580 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:08:18.396426 28595 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
W20240501 14:08:18.396890 30861 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 -> Peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Couldn't send request to peer 3a3972b170c54344a4ba61f176f837ba. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:18.398128 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:18.399036 29136 raft_consensus.cc:1261] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:18.399587 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet efe7fed8e5d44e8a80ffbdc27619a74d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 14:08:18.400692 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:18.400640 16546 consensus_queue.cc:1035] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:18.400843 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:08:18.400766 16580 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.401403 16565 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:18.402128 29006 raft_consensus.cc:2932] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.404032 16546 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:18.407014 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 60b3db79ff754e8bbe9e5865f7104312 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:18.409401 29136 raft_consensus.cc:2932] T 111ba172d7e44054872c8fc8cac8ba88 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:18.410269 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 111ba172d7e44054872c8fc8cac8ba88 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:18.408927 28595 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 11 to 12, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New cstate: current_term: 2 leader_uuid: "a0e901103604414c829b40edf9f24c28" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:18.419370 28596 catalog_manager.cc:5484] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:18.425868 29136 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:08:18.426942 16583 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:08:18.431424 29006 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:08:18.432315 16587 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:08:18.451839 16587 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:18.457573 29007 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:18.469108 28599 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
W20240501 14:08:18.470561 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:18.488418 28745 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:08:18.491050 28679 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b473e10a8ad4bd58bb44aabd9656e08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:18.491029 29006 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:18.491725 16582 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:18.495267 16158 raft_consensus.cc:998] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.469s)
I20240501 14:08:18.496726 16158 consensus_queue.cc:237] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:18.496567 16583 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.498020 16587 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.502528 29006 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.504168 28599 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 8 to 9, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.510484 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9b473e10a8ad4bd58bb44aabd9656e08 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:18.515112 28595 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 7, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.528699 29136 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.545002 30793 consensus_queue.cc:237] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:18.546343 29006 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:18.547058 16158 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:18.548992 29136 raft_consensus.cc:1261] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Refusing update from remote peer 8fc5d8222c164a679e7a99966da1eaaf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:08:18.550406 16583 consensus_queue.cc:1035] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:08:18.550879 30793 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:08:18.551155 30728 consensus_peers.cc:485] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Couldn't send request to peer cd3c04ebb6c84b15a1b3adc748737094. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:18.551543 16158 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:18.555980 30793 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.559371 16583 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:18.563442 29136 raft_consensus.cc:2932] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:18.564966 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 14:08:18.565559 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:18.567775 28599 catalog_manager.cc:5484] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 9 to 10, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New cstate: current_term: 2 leader_uuid: "8fc5d8222c164a679e7a99966da1eaaf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:18.579169 16597 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.583284 28599 catalog_manager.cc:5484] T 6b6a54ac32694640bffdf1c7f3b035cc P 8fc5d8222c164a679e7a99966da1eaaf reported cstate change: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:18.583292 29006 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
W20240501 14:08:18.653592 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:08:18.655303 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:18.754810 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:18.772893 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:18.775262 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:18.790089 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:18.796216 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:08:18.797382 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:18.803844 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:18.813130 16651 ts_tablet_manager.cc:902] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Initiating tablet copy from peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:08:18.826351 16651 tablet_copy_client.cc:308] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.135:42127
I20240501 14:08:18.826933 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:18.827283 30948 tablet_copy_service.cc:140] P a0e901103604414c829b40edf9f24c28: Received BeginTabletCopySession request for tablet 60b3db79ff754e8bbe9e5865f7104312 from peer 3a3972b170c54344a4ba61f176f837ba ({username='slave'} at 127.16.36.132:45273)
I20240501 14:08:18.827461 30948 tablet_copy_service.cc:161] P a0e901103604414c829b40edf9f24c28: Beginning new tablet copy session on tablet 60b3db79ff754e8bbe9e5865f7104312 from peer 3a3972b170c54344a4ba61f176f837ba at {username='slave'} at 127.16.36.132:45273: session id = 3a3972b170c54344a4ba61f176f837ba-60b3db79ff754e8bbe9e5865f7104312
I20240501 14:08:18.828112 30775 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:18.831636 30948 tablet_copy_source_session.cc:205] T 60b3db79ff754e8bbe9e5865f7104312 P a0e901103604414c829b40edf9f24c28: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:18.838742 16651 ts_tablet_manager.cc:1875] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:08:18.881688 16651 ts_tablet_manager.cc:1888] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240501 14:08:18.882113 16651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b3db79ff754e8bbe9e5865f7104312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:18.905738 16651 tablet_copy_client.cc:783] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:18.922421 16651 tablet_copy_client.cc:647] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:18.933921 16658 ts_tablet_manager.cc:902] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
W20240501 14:08:18.942504 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:18.943571 16651 tablet_copy_client.cc:523] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:08:18.985626 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:18.990607 16651 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:08:18.995214 16662 ts_tablet_manager.cc:902] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
W20240501 14:08:19.012280 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:19.012547 16662 tablet_copy_client.cc:308] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:08:19.013340 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet efe7fed8e5d44e8a80ffbdc27619a74d from peer 3a3972b170c54344a4ba61f176f837ba ({username='slave'} at 127.16.36.132:37815)
I20240501 14:08:19.013511 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet efe7fed8e5d44e8a80ffbdc27619a74d from peer 3a3972b170c54344a4ba61f176f837ba at {username='slave'} at 127.16.36.132:37815: session id = 3a3972b170c54344a4ba61f176f837ba-efe7fed8e5d44e8a80ffbdc27619a74d
I20240501 14:08:19.017325 29027 tablet_copy_source_session.cc:205] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:19.019166 16658 tablet_copy_client.cc:308] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:08:19.019212 16662 ts_tablet_manager.cc:1875] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:08:19.026384 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 111ba172d7e44054872c8fc8cac8ba88 from peer cd3c04ebb6c84b15a1b3adc748737094 ({username='slave'} at 127.16.36.130:47511)
I20240501 14:08:19.026552 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 111ba172d7e44054872c8fc8cac8ba88 from peer cd3c04ebb6c84b15a1b3adc748737094 at {username='slave'} at 127.16.36.130:47511: session id = cd3c04ebb6c84b15a1b3adc748737094-111ba172d7e44054872c8fc8cac8ba88
I20240501 14:08:19.030234 29027 tablet_copy_source_session.cc:205] T 111ba172d7e44054872c8fc8cac8ba88 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:19.032536 16658 ts_tablet_manager.cc:1875] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:08:19.036285 16671 ts_tablet_manager.cc:902] T 9b473e10a8ad4bd58bb44aabd9656e08 P 8fc5d8222c164a679e7a99966da1eaaf: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
F20240501 14:08:19.036924 16671 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.queue_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.queue_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_cpu_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_cpu_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_wall_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_wall_time_us) (tablet-download-pool-2020fe37059a4e74bb45190398e4e177.queue_time_us, tablet-download-pool-2020fe37059a4e74bb45190398e4e177.queue_time_us) (tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_cpu_time_us, tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_cpu_time_us) (tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_wall_time_us, tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_wall_time_us) (tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.queue_time_us, tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.queue_time_us) (tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_cpu_time_us, tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_cpu_time_us) (tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_wall_time_us, tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_wall_time_us) (tablet-download-pool-39f1ab3a94db41b08ac90bba52418a93.queue_time_us, tablet-download-pool-39f1ab3a94db41b08ac90bba52418a93.queue_time_us) (tablet-download-pool-39f1ab3a94db41b08ac90bba52418a93.run_cpu_time_us, tablet-download-pool-39f1ab3a94db41b08ac90bba52418a93.run_cpu_time_us) (tablet-download-pool-39f1ab3a94db41b08ac90bba52418a93.run_wall_time_us, tablet-download-pool-39f1ab3a94db41b08ac90bba52418a93.run_wall_time_us) (tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.queue_time_us, tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.queue_time_us) (tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.run_cpu_time_us, tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.run_cpu_time_us) (tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.run_wall_time_us, tablet-download-pool-4d8c435795a04b1b90b6e2ff914676c8.run_wall_time_us) (tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.queue_time_us, tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.queue_time_us) (tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_cpu_time_us, tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_cpu_time_us) (tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_wall_time_us, tablet-download-pool-63b7f2a8d8a84faf86c4de16b3cc6663.run_wall_time_us) (tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.queue_time_us, tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.queue_time_us) (tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_cpu_time_us, tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_cpu_time_us) (tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_wall_time_us, tablet-download-pool-6b6a54ac32694640bffdf1c7f3b035cc.run_wall_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.queue_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.queue_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_cpu_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_cpu_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_wall_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_wall_time_us) (tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.queue_time_us, tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.queue_time_us) (tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_cpu_time_us, tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_cpu_time_us) (tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_wall_time_us, tablet-download-pool-763b2e84afa64924ad5eeeeb3265a1d8.run_wall_time_us) (tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.queue_time_us, tablet-download-pool-9b473e10a8ad4bd58bb44aabd9656e08.queue_time_us) (tablet-download-pool-bb712628c7764a138f5425d14b724d9e.queue_time_us, tablet-download-pool-bb712628c7764a138f5425d14b724d9e.queue_time_us) (tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_cpu_time_us, tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_cpu_time_us) (tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_wall_time_us, tablet-download-pool-bb712628c7764a138f5425d14b724d9e.run_wall_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.queue_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.queue_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_cpu_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_cpu_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_wall_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_wall_time_us) (tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.queue_time_us, tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.queue_time_us) (tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_cpu_time_us, tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_cpu_time_us) (tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_wall_time_us, tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_wall_time_us) (tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.queue_time_us, tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.queue_time_us) (tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_cpu_time_us, tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_cpu_time_us) (tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_wall_time_us, tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.run_wall_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.queue_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.queue_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.run_cpu_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.run_cpu_time_us) (tablet-download-pool-f4402de0b48041c383568feb9365147e.run_wall_time_us, tablet-download-pool-f4402de0b48041c383568feb9365147e.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f9844348dcd google::LogMessage::Fail() at ??:0
I20240501 14:08:19.043429 16658 ts_tablet_manager.cc:1888] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:08:19.043802 16658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111ba172d7e44054872c8fc8cac8ba88. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f984434cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f98443487cc google::LogMessage::Flush() at ??:0
@ 0x7f9844349f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:08:19.049069 16658 tablet_copy_client.cc:783] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 5 data blocks...
@ 0x7f9847089742 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:08:19.057453 16658 tablet_copy_client.cc:647] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:19.059809 16658 tablet_copy_client.cc:523] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:19.067548 16658 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
@ 0x7f9847055f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f984704fcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 14:08:19.086238 16662 ts_tablet_manager.cc:1888] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:08:19.086658 16662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe7fed8e5d44e8a80ffbdc27619a74d. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f98564a28de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f98564badaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:08:19.114364 16582 raft_consensus.cc:998] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Attempting to remove follower 826ccf8358b6453d88bcb132290b82a5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 826ccf8358b6453d88bcb132290b82a5 for more than 300 seconds (300.068s)
I20240501 14:08:19.115182 16582 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:19.125908 16658 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:19.126585 16658 tablet_bootstrap.cc:492] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap complete.
I20240501 14:08:19.127013 16658 ts_tablet_manager.cc:1367] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.060s user 0.033s sys 0.021s
I20240501 14:08:19.128116 16658 raft_consensus.cc:348] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:19.128461 16658 raft_consensus.cc:729] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:08:19.128809 16658 consensus_queue.cc:260] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:19.131141 16658 ts_tablet_manager.cc:1398] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
@ 0x7f98565e5480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:08:19.132047 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session cd3c04ebb6c84b15a1b3adc748737094-111ba172d7e44054872c8fc8cac8ba88 received from {username='slave'} at 127.16.36.130:47511
I20240501 14:08:19.132179 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session cd3c04ebb6c84b15a1b3adc748737094-111ba172d7e44054872c8fc8cac8ba88 on tablet 111ba172d7e44054872c8fc8cac8ba88 with peer cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:19.136248 16662 tablet_copy_client.cc:783] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 5 data blocks...
W20240501 14:08:19.141332 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b473e10a8ad4bd58bb44aabd9656e08: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:19.145262 16662 tablet_copy_client.cc:647] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 1 WAL segments...
@ 0x7f985661da8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 14:08:19.147552 16662 tablet_copy_client.cc:523] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:19.150523 28875 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:08:19.151317 16582 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:08:19.153038 16662 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
@ 0x7f9856600981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 14:08:19.166795 16682 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:19.167853 16658 ts_tablet_manager.cc:902] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Initiating tablet copy from peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
@ 0x7f985660070e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:08:19.175359 28875 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
@ 0x7f9847059b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f984705d19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f984705d019 std::_Function_handler<>::_M_invoke() at ??:0
I20240501 14:08:19.197990 28603 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 5 to 6, VOTER 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:19.202642 16658 tablet_copy_client.cc:308] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.134:44241
I20240501 14:08:19.203842 30816 tablet_copy_service.cc:140] P 8fc5d8222c164a679e7a99966da1eaaf: Received BeginTabletCopySession request for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 from peer cd3c04ebb6c84b15a1b3adc748737094 ({username='slave'} at 127.16.36.130:55461)
I20240501 14:08:19.204021 30816 tablet_copy_service.cc:161] P 8fc5d8222c164a679e7a99966da1eaaf: Beginning new tablet copy session on tablet 63b7f2a8d8a84faf86c4de16b3cc6663 from peer cd3c04ebb6c84b15a1b3adc748737094 at {username='slave'} at 127.16.36.130:55461: session id = cd3c04ebb6c84b15a1b3adc748737094-63b7f2a8d8a84faf86c4de16b3cc6663
I20240501 14:08:19.208392 30816 tablet_copy_source_session.cc:205] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 8fc5d8222c164a679e7a99966da1eaaf: Tablet Copy: opened 5 blocks and 1 log segments
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:08:19.210114 16658 ts_tablet_manager.cc:1875] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f984703801e kudu::Thread::SuperviseThread() at ??:0
I20240501 14:08:19.220507 16662 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:19.221997 16658 ts_tablet_manager.cc:1888] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 14:08:19.222311 16658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7f2a8d8a84faf86c4de16b3cc6663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:19.223317 16662 tablet_bootstrap.cc:492] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Bootstrap complete.
I20240501 14:08:19.223680 16662 ts_tablet_manager.cc:1367] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.071s user 0.033s sys 0.004s
W20240501 14:08:19.225759 30730 consensus_peers.cc:485] T 1fdedd39e12a49659d989f37b3287d87 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:08:19.224556 16662 raft_consensus.cc:348] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:19.225975 16662 raft_consensus.cc:729] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:08:19.226245 16662 consensus_queue.cc:260] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:19.226843 16662 ts_tablet_manager.cc:1398] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:19.229626 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 3a3972b170c54344a4ba61f176f837ba-efe7fed8e5d44e8a80ffbdc27619a74d received from {username='slave'} at 127.16.36.132:37815
I20240501 14:08:19.229769 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 3a3972b170c54344a4ba61f176f837ba-efe7fed8e5d44e8a80ffbdc27619a74d on tablet efe7fed8e5d44e8a80ffbdc27619a74d with peer 3a3972b170c54344a4ba61f176f837ba
I20240501 14:08:19.231150 16658 tablet_copy_client.cc:783] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 5 data blocks...
W20240501 14:08:19.234216 30730 consensus_peers.cc:462] T 4d8c435795a04b1b90b6e2ff914676c8 P 8fc5d8222c164a679e7a99966da1eaaf -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
@ 0x7f984a3bd6db start_thread at ??:0
@ 0x7f98431fb71f clone at ??:0
I20240501 14:08:19.244225 16651 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:19.244802 16651 tablet_bootstrap.cc:492] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap complete.
I20240501 14:08:19.245154 16651 ts_tablet_manager.cc:1367] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.255s user 0.029s sys 0.007s
I20240501 14:08:19.246065 16651 raft_consensus.cc:348] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:19.246322 16651 raft_consensus.cc:729] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:08:19.267155 16651 consensus_queue.cc:260] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:19.271036 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:19.275106 16651 ts_tablet_manager.cc:1398] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.030s user 0.001s sys 0.002s
I20240501 14:08:19.275852 30948 tablet_copy_service.cc:342] P a0e901103604414c829b40edf9f24c28: Request end of tablet copy session 3a3972b170c54344a4ba61f176f837ba-60b3db79ff754e8bbe9e5865f7104312 received from {username='slave'} at 127.16.36.132:45273
I20240501 14:08:19.275974 30948 tablet_copy_service.cc:433] P a0e901103604414c829b40edf9f24c28: ending tablet copy session 3a3972b170c54344a4ba61f176f837ba-60b3db79ff754e8bbe9e5865f7104312 on tablet 60b3db79ff754e8bbe9e5865f7104312 with peer 3a3972b170c54344a4ba61f176f837ba
I20240501 14:08:19.306263 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:19.318302 29136 raft_consensus.cc:1203] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 14:08:19.335649 28745 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:19.338708 28875 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:19.338892 29136 raft_consensus.cc:1203] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:08:19.339565 16682 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:19.342778 16682 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:08:19.347103 28875 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
W20240501 14:08:19.354359 28679 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 -> Peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): Couldn't send request to peer 7e7bc120bd74479ab378f779778fd1e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 135da52054584a27959efdd9126cfcef. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:19.355309 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 135da52054584a27959efdd9126cfcef with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:19.357887 28599 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 6 to 7, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:08:19.371052 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:19.378324 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(111ba172d7e44054872c8fc8cac8ba88): 4922 bytes on disk
I20240501 14:08:19.379050 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(111ba172d7e44054872c8fc8cac8ba88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240501 14:08:19.399381 28875 raft_consensus.cc:1203] T 111ba172d7e44054872c8fc8cac8ba88 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:08:19.433650 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:19.435036 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:19.463109 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(60b3db79ff754e8bbe9e5865f7104312): 4921 bytes on disk
I20240501 14:08:19.463804 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(60b3db79ff754e8bbe9e5865f7104312) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:08:19.464964 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(efe7fed8e5d44e8a80ffbdc27619a74d): 4834 bytes on disk
I20240501 14:08:19.465433 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(efe7fed8e5d44e8a80ffbdc27619a74d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 14:08:19.508343 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:08:19.563155 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:19.687494 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:19.689584 28679 connection.cc:535] client connection to 127.16.36.134:44241 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:08:19.694173 28941 connection.cc:535] client connection to 127.16.36.134:44241 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:08:19.697278 28810 connection.cc:535] client connection to 127.16.36.134:44241 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:08:19.698997 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:19.700837 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:19.708108 29072 connection.cc:535] client connection to 127.16.36.134:44241 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:08:19.712181 13607 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:08:19.690732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.134:44241 (local address 127.16.36.129:33875)
0501 14:08:19.691021 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0501 14:08:19.691040 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0501 14:08:19.691193 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:08:19.711554 (+ 20361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":175}
W20240501 14:08:19.712546 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:08:19.712687 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:19.713632 16570 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:08:19.695181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.134:44241 (local address 127.16.36.131:51825)
0501 14:08:19.695332 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 14:08:19.695347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 14:08:19.695439 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:08:19.713147 (+ 17708us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":58}
W20240501 14:08:19.713889 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:19.714910 16391 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:08:19.698903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.134:44241 (local address 127.16.36.130:46269)
0501 14:08:19.702849 (+ 3946us) negotiation.cc:107] Waiting for socket to connect
0501 14:08:19.702877 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 14:08:19.703044 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:08:19.714312 (+ 11268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3760}
W20240501 14:08:19.715214 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:19.715404 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:19.715831 16658 ts_tablet_manager.cc:980] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: Unable to fetch data from remote peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Unable to download block with id 7679428818038437264: Unable to download block 7679428818038437264: unable to fetch data from remote: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:08:19.716002 16658 tablet_replica.cc:330] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: stopping tablet replica
I20240501 14:08:19.716104 16658 raft_consensus.cc:2229] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:19.716243 16658 raft_consensus.cc:2258] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 14:08:19.717883 16658 tablet_copy_client.cc:1108] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:19.718717 16658 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Already present: tried to load directory group for tablet 63b7f2a8d8a84faf86c4de16b3cc6663 but one is already registered
I20240501 14:08:19.718923 16658 ts_tablet_manager.cc:1875] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:08:19.738567 15900 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:08:19.710320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.134:44241 (local address 127.16.36.132:48441)
0501 14:08:19.710507 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0501 14:08:19.710528 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 14:08:19.710686 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:08:19.737894 (+ 27208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":50}
W20240501 14:08:19.738818 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:19.738943 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:19.738970 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:19.753850 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:19.760816 16658 ts_tablet_manager.cc:1888] T 63b7f2a8d8a84faf86c4de16b3cc6663 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
W20240501 14:08:19.801614 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:19.877070 16775 ts_tablet_manager.cc:902] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:08:19.878698 16775 tablet_copy_client.cc:308] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:08:19.879498 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 135da52054584a27959efdd9126cfcef from peer 7e7bc120bd74479ab378f779778fd1e8 ({username='slave'} at 127.16.36.131:39287)
I20240501 14:08:19.879686 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 135da52054584a27959efdd9126cfcef from peer 7e7bc120bd74479ab378f779778fd1e8 at {username='slave'} at 127.16.36.131:39287: session id = 7e7bc120bd74479ab378f779778fd1e8-135da52054584a27959efdd9126cfcef
I20240501 14:08:19.883705 28765 tablet_copy_source_session.cc:205] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:19.885538 16775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135da52054584a27959efdd9126cfcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:19.893337 16775 tablet_copy_client.cc:783] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:19.902877 16775 tablet_copy_client.cc:647] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:19.904767 16775 tablet_copy_client.cc:523] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:19.908280 16775 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
I20240501 14:08:19.928694 16775 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:19.929236 16775 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap complete.
I20240501 14:08:19.929591 16775 ts_tablet_manager.cc:1367] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 14:08:19.930446 16775 raft_consensus.cc:348] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:19.930711 16775 raft_consensus.cc:729] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:08:19.930981 16775 consensus_queue.cc:260] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:19.931600 16775 ts_tablet_manager.cc:1398] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:08:19.932524 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 7e7bc120bd74479ab378f779778fd1e8-135da52054584a27959efdd9126cfcef received from {username='slave'} at 127.16.36.131:39287
I20240501 14:08:19.932672 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 7e7bc120bd74479ab378f779778fd1e8-135da52054584a27959efdd9126cfcef on tablet 135da52054584a27959efdd9126cfcef with peer 7e7bc120bd74479ab378f779778fd1e8
W20240501 14:08:20.017809 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.060061 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:20.066784 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:20.100715 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:20.109062 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:20.121085 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef): 4921 bytes on disk
I20240501 14:08:20.122074 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4}
I20240501 14:08:20.141945 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:20.167745 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:20.271308 29006 raft_consensus.cc:1203] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:08:20.283334 16580 raft_consensus.cc:482] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.283535 16580 raft_consensus.cc:504] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:20.284442 16580 leader_election.cc:288] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:20.335315 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:20.339354 30862 leader_election.cc:334] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.339794 30862 leader_election.cc:302] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:20.340255 16580 raft_consensus.cc:2726] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:20.361927 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:20.378376 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:20.393154 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fb04ec77da194832aa63d69aba307de3: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:20.396772 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:08:20.412067 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:20.428946 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:20.434394 16762 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.434720 16762 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:20.435726 16762 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:08:20.441007 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:20.446506 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:20.447193 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.449352 16762 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:20.463090 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:20.475169 28856 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:20.476975 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 14:08:20.494251 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:20.508795 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:08:20.556780 16814 raft_consensus.cc:482] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.557042 16814 raft_consensus.cc:504] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:20.568940 16580 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.569237 16580 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:20.570608 16580 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:08:20.571529 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
W20240501 14:08:20.573097 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:20.573218 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 3a3972b170c54344a4ba61f176f837ba
W20240501 14:08:20.573405 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.573709 16580 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:20.576907 16814 leader_election.cc:288] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:08:20.578933 28810 leader_election.cc:334] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.582720 16762 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.582928 16762 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:20.583901 16762 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:20.584025 16682 raft_consensus.cc:482] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.584223 16682 raft_consensus.cc:504] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:20.585101 16682 leader_election.cc:288] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:08:20.586241 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:20.587318 28679 leader_election.cc:334] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.590112 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:20.600775 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:20.601094 30925 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 2.
I20240501 14:08:20.601365 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
I20240501 14:08:20.602244 28680 leader_election.cc:302] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144; no voters: 8fc5d8222c164a679e7a99966da1eaaf, a0e901103604414c829b40edf9f24c28
W20240501 14:08:20.602609 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:20.602912 16682 raft_consensus.cc:2726] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:20.602918 28811 leader_election.cc:302] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0e901103604414c829b40edf9f24c28, cd3c04ebb6c84b15a1b3adc748737094; no voters: 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.604027 16878 raft_consensus.cc:2781] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:08:20.604130 16878 raft_consensus.cc:482] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.604228 16878 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:08:20.604668 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:08:20.604806 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:20.605315 16762 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:20.610201 16878 raft_consensus.cc:504] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:20.611006 16878 leader_election.cc:288] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 election: Requested vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:08:20.614041 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2020fe37059a4e74bb45190398e4e177" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28"
I20240501 14:08:20.614295 30925 raft_consensus.cc:3037] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:08:20.616658 30925 raft_consensus.cc:2454] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 3.
W20240501 14:08:20.612524 28810 leader_election.cc:334] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.617866 28811 leader_election.cc:302] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0e901103604414c829b40edf9f24c28, cd3c04ebb6c84b15a1b3adc748737094; no voters: 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.618221 16814 raft_consensus.cc:2781] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:08:20.618383 16814 raft_consensus.cc:686] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [term 3 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:08:20.618716 16814 consensus_queue.cc:237] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:20.622584 28598 catalog_manager.cc:5484] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 2 to 3, leader changed from 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 3 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:20.642347 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:20.650310 28986 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:08:20.653805 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:20.664145 16580 raft_consensus.cc:482] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.664364 16580 raft_consensus.cc:504] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:20.665195 16580 leader_election.cc:288] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:08:20.668166 16682 raft_consensus.cc:482] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.668413 16682 raft_consensus.cc:504] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:20.669355 16682 leader_election.cc:288] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:08:20.671123 30862 leader_election.cc:334] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.671780 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "343a85ba3b514663af5320cd14a9e144" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
I20240501 14:08:20.684937 16737 raft_consensus.cc:482] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.685218 16737 raft_consensus.cc:504] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:20.686690 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:08:20.686926 28743 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 2.
I20240501 14:08:20.687151 16737 leader_election.cc:288] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:08:20.687762 28942 leader_election.cc:302] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:08:20.689082 16737 raft_consensus.cc:2781] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:08:20.689216 16737 raft_consensus.cc:482] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.689339 16737 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:08:20.694165 28941 leader_election.cc:334] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.702390 16737 raft_consensus.cc:504] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:20.704982 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f1ab3a94db41b08ac90bba52418a93" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:08:20.705243 28743 raft_consensus.cc:3037] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:08:20.706597 16737 leader_election.cc:288] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 election: Requested vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:08:20.707489 28941 leader_election.cc:334] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.707918 16762 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.708137 16762 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:20.709697 28743 raft_consensus.cc:2454] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 3.
I20240501 14:08:20.710247 28942 leader_election.cc:302] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, 7e7bc120bd74479ab378f779778fd1e8; no voters: 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.710872 16737 raft_consensus.cc:2781] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:08:20.717096 30907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:20.719796 16737 raft_consensus.cc:686] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:08:20.720161 16737 consensus_queue.cc:237] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:20.725085 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144" is_pre_election: true
I20240501 14:08:20.725337 28743 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
I20240501 14:08:20.725744 30863 leader_election.cc:302] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, a0e901103604414c829b40edf9f24c28; no voters: 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.726262 16580 raft_consensus.cc:2781] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:08:20.726366 16580 raft_consensus.cc:482] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:20.726469 16580 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:08:20.727140 28679 leader_election.cc:334] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.727314 28679 leader_election.cc:302] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144; no voters: 7e7bc120bd74479ab378f779778fd1e8, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.727684 16682 raft_consensus.cc:2726] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:20.730396 16762 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:08:20.731362 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:20.731627 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
I20240501 14:08:20.732750 16580 raft_consensus.cc:504] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
W20240501 14:08:20.733871 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:20.735903 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:08:20.736936 30862 leader_election.cc:334] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.738411 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1b3fec785f43c4a3bacedecd8472dd" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:08:20.738674 28743 raft_consensus.cc:3037] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:08:20.742710 28743 raft_consensus.cc:2454] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 3.
I20240501 14:08:20.738152 16580 leader_election.cc:288] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 election: Requested vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:08:20.743870 30863 leader_election.cc:302] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144, a0e901103604414c829b40edf9f24c28; no voters: 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.744599 16580 raft_consensus.cc:2781] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:08:20.756563 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:20.771131 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.771338 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:20.772373 16580 raft_consensus.cc:686] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [term 3 LEADER]: Becoming Leader. State: Replica: a0e901103604414c829b40edf9f24c28, State: Running, Role: LEADER
I20240501 14:08:20.772660 16762 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:20.773361 16580 consensus_queue.cc:237] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:20.781777 28598 catalog_manager.cc:5484] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 2 to 3, leader changed from 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 3 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:20.786594 28604 catalog_manager.cc:5484] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 reported cstate change: term changed from 2 to 3, leader changed from 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) to a0e901103604414c829b40edf9f24c28 (127.16.36.135). New cstate: current_term: 3 leader_uuid: "a0e901103604414c829b40edf9f24c28" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
W20240501 14:08:20.878898 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:20.986932 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:20.992772 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:21.012789 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:21.017619 30925 raft_consensus.cc:1261] T 2020fe37059a4e74bb45190398e4e177 P a0e901103604414c829b40edf9f24c28 [term 3 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 14:08:21.020246 16814 consensus_queue.cc:1035] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:08:21.022356 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:21.039762 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:21.053726 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:21.060585 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:21.106236 28743 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 14:08:21.107334 16895 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:08:21.115850 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:21.229060 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:21.244980 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:21.254513 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:21.257927 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:21.269163 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:21.270759 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:21.273452 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:21.288208 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:21.325278 28743 raft_consensus.cc:1261] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Refusing update from remote peer a0e901103604414c829b40edf9f24c28: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 14:08:21.325958 16580 consensus_queue.cc:1035] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:08:21.386904 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:21.574110 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:21.574801 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:21.597788 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:21.599290 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:21.635069 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:21.653337 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:21.681483 16879 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:08:21.681787 16879 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
W20240501 14:08:21.684337 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:21.684439 16879 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:08:21.687108 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:21.687855 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:21.688009 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:08:21.688954 16879 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:21.711707 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 14:08:21.715827 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:21.723544 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:21.734578 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:08:21.737072 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:21.737447 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:21.748492 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:21.763229 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:21.769670 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:21.801074 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:21.816787 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:08:21.883574 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:21.918732 28743 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "135da52054584a27959efdd9126cfcef"
dest_uuid: "343a85ba3b514663af5320cd14a9e144"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56562
I20240501 14:08:21.918994 28743 raft_consensus.cc:595] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:08:21.956439 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:08:21.981431 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:22.049381 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:22.055105 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:22.061934 28723 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:08:22.079254 28856 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:22.148514 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:22.202219 16682 raft_consensus.cc:979] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: : Instructing follower cd3c04ebb6c84b15a1b3adc748737094 to start an election
I20240501 14:08:22.202389 16682 raft_consensus.cc:1067] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Signalling peer cd3c04ebb6c84b15a1b3adc748737094 to start an election
I20240501 14:08:22.203258 28875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "135da52054584a27959efdd9126cfcef"
dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094"
from {username='slave'} at 127.16.36.129:56137
I20240501 14:08:22.203469 28875 raft_consensus.cc:482] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:08:22.203565 28875 raft_consensus.cc:3037] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:22.206609 28875 raft_consensus.cc:504] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:22.207527 28875 leader_election.cc:288] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 2 election: Requested vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:08:22.208042 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135da52054584a27959efdd9126cfcef" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "343a85ba3b514663af5320cd14a9e144"
I20240501 14:08:22.208127 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135da52054584a27959efdd9126cfcef" candidate_uuid: "cd3c04ebb6c84b15a1b3adc748737094" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "7e7bc120bd74479ab378f779778fd1e8"
I20240501 14:08:22.208256 28743 raft_consensus.cc:3032] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:08:22.208329 28743 raft_consensus.cc:729] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Running, Role: LEADER
I20240501 14:08:22.208356 29006 raft_consensus.cc:3037] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:22.208582 28743 consensus_queue.cc:260] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:22.208922 28743 raft_consensus.cc:3037] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:22.211635 28743 raft_consensus.cc:2454] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 2.
I20240501 14:08:22.211747 29006 raft_consensus.cc:2454] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3c04ebb6c84b15a1b3adc748737094 in term 2.
I20240501 14:08:22.212209 28810 leader_election.cc:302] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e7bc120bd74479ab378f779778fd1e8, cd3c04ebb6c84b15a1b3adc748737094; no voters:
I20240501 14:08:22.212558 17057 raft_consensus.cc:2781] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:22.212718 17057 raft_consensus.cc:686] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 2 LEADER]: Becoming Leader. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Running, Role: LEADER
I20240501 14:08:22.212972 17057 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:22.215989 28595 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: term changed from 1 to 2, leader changed from 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) to cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130). New cstate: current_term: 2 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
W20240501 14:08:22.227733 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:08:22.310901 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:08:22.319991 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:22.330755 16762 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:22.330967 16762 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:22.331980 16762 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:08:22.333403 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:22.333649 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
I20240501 14:08:22.334462 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:22.338057 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:22.338379 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:22.338536 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:22.339296 16762 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:22.356297 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:22.372591 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:22.374497 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:22.398461 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:22.400904 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:22.426316 16762 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:22.426550 16762 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:22.427310 16762 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:08:22.429718 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:22.439301 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:22.439584 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:22.439901 16762 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:22.622581 16762 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:22.622840 16762 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:22.623601 16762 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:22.624581 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:22.625262 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:22.625437 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:08:22.625543 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:22.627959 17190 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:22.635977 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:22.638105 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:22.649938 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:22.651899 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:22.652078 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:22.671214 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 60b3db79ff754e8bbe9e5865f7104312: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:22.700551 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:08:22.732478 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:22.736457 28743 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:08:22.737007 17057 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:08:22.774580 29006 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:08:22.775498 17205 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:08:22.809651 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:08:22.894577 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:22.959496 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:22.973385 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:22.977222 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:22.992633 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:23.007048 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:23.032691 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:23.041648 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:23.044489 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:08:23.050902 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:08:23.061081 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:23.082518 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:23.129652 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:23.154341 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 56 similar messages]
I20240501 14:08:23.225071 28875 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:08:23.227020 29006 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:08:23.227643 17057 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:08:23.229866 17205 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:08:23.230556 29006 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:08:23.237187 28598 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 7 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) evicted. New cstate: current_term: 2 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:08:23.242400 28723 tablet_service.cc:1514] Processing DeleteTablet for tablet 135da52054584a27959efdd9126cfcef with delete_type TABLET_DATA_TOMBSTONED (TS 343a85ba3b514663af5320cd14a9e144 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56374
I20240501 14:08:23.243177 17296 tablet_replica.cc:330] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: stopping tablet replica
I20240501 14:08:23.243557 17296 raft_consensus.cc:2229] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:23.243798 17296 raft_consensus.cc:2258] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:23.245219 17296 ts_tablet_manager.cc:1875] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:08:23.250010 28875 consensus_queue.cc:237] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:23.253185 29006 raft_consensus.cc:1261] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:08:23.254022 17205 consensus_queue.cc:1035] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:08:23.259400 17057 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
W20240501 14:08:23.261382 28809 consensus_peers.cc:485] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Couldn't send request to peer 3a3972b170c54344a4ba61f176f837ba. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:23.264544 29006 raft_consensus.cc:2932] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:23.274873 28598 catalog_manager.cc:5484] T 135da52054584a27959efdd9126cfcef P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 9 to 10, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New cstate: current_term: 2 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:08:23.276087 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 135da52054584a27959efdd9126cfcef with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:23.278302 17296 ts_tablet_manager.cc:1888] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:08:23.278470 17296 log.cc:1196] T 135da52054584a27959efdd9126cfcef P 343a85ba3b514663af5320cd14a9e144: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/wals/135da52054584a27959efdd9126cfcef
I20240501 14:08:23.279498 28590 catalog_manager.cc:4830] TS 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): tablet 135da52054584a27959efdd9126cfcef (table rebalance_test_table_6 [id=6d518ad3360047ccb71449199dc94b10]) successfully deleted
W20240501 14:08:23.290160 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:23.514165 28680 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:23.546922 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:23.551102 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:23.553321 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:23.570767 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:23.573830 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:23.623708 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:23.641813 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:08:23.661806 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 14:08:23.677778 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:23.693945 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240501 14:08:23.695247 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:23.724545 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:23.741693 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:23.746798 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:23.749478 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:23.749796 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:23.781060 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:23.822441 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:08:23.863142 17396 ts_tablet_manager.cc:902] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:23.864558 17396 tablet_copy_client.cc:308] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:08:23.865397 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 135da52054584a27959efdd9126cfcef from peer 3a3972b170c54344a4ba61f176f837ba ({username='slave'} at 127.16.36.132:45291)
I20240501 14:08:23.865572 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 135da52054584a27959efdd9126cfcef from peer 3a3972b170c54344a4ba61f176f837ba at {username='slave'} at 127.16.36.132:45291: session id = 3a3972b170c54344a4ba61f176f837ba-135da52054584a27959efdd9126cfcef
I20240501 14:08:23.868983 28896 tablet_copy_source_session.cc:205] T 135da52054584a27959efdd9126cfcef P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:23.870501 17396 ts_tablet_manager.cc:1875] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:08:23.878086 17396 ts_tablet_manager.cc:1888] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:08:23.878450 17396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135da52054584a27959efdd9126cfcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:23.882328 17396 tablet_copy_client.cc:783] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:23.890007 17396 tablet_copy_client.cc:647] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:23.892015 17396 tablet_copy_client.cc:523] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:23.897006 17396 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:08:23.960438 17396 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:23.962738 17396 tablet_bootstrap.cc:492] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Bootstrap complete.
I20240501 14:08:23.963070 17396 ts_tablet_manager.cc:1367] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.066s user 0.062s sys 0.001s
I20240501 14:08:23.964630 17190 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:23.965920 17190 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:23.966589 17396 raft_consensus.cc:348] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:23.967089 17396 raft_consensus.cc:729] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:08:23.967329 17396 consensus_queue.cc:260] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:08:23.969803 17190 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:08:23.971117 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:23.971410 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:08:23.973729 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:23.974092 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:08:23.974396 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:23.974507 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:23.974954 17396 ts_tablet_manager.cc:1398] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20240501 14:08:23.976621 17430 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:23.976758 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 3a3972b170c54344a4ba61f176f837ba-135da52054584a27959efdd9126cfcef received from {username='slave'} at 127.16.36.132:45291
I20240501 14:08:23.976926 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 3a3972b170c54344a4ba61f176f837ba-135da52054584a27959efdd9126cfcef on tablet 135da52054584a27959efdd9126cfcef with peer 3a3972b170c54344a4ba61f176f837ba
I20240501 14:08:23.994210 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef): 4921 bytes on disk
I20240501 14:08:23.995406 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(135da52054584a27959efdd9126cfcef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 14:08:23.999950 17430 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:24.000123 17430 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:24.000949 17430 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:08:24.002185 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:24.002847 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:24.002962 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:24.003415 17430 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:24.054234 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 47268.1us
W20240501 14:08:24.056476 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:08:24.085954 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:24.099249 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:24.108059 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:24.148885 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:24.152271 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:24.158638 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:24.171788 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:24.207280 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:24.215103 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:08:24.216810 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:24.225062 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:24.232367 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:08:24.232681 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:24.239288 29136 raft_consensus.cc:1203] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240501 14:08:24.257526 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:24.283528 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:24.291018 17430 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:24.291230 17430 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:24.291939 17430 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:24.294124 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:24.294836 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:08:24.303737 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:24.304585 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:24.338485 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:24.338662 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:24.339200 17430 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:24.377393 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:08:24.580037 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 14:08:24.608382 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:08:24.629380 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:08:24.663830 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:08:24.676358 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:24.706317 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:24.710649 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:24.738057 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:24.750619 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:24.775003 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:24.803304 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:24.910131 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:24.916123 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:24.925871 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:24.939666 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:24.948525 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:25.078938 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:08:25.134699 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:25.296224 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:25.297302 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:25.297626 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:25.313386 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:25.326026 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:25.334931 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:25.419085 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:08:25.463618 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:08:25.464766 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:08:25.511023 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:25.522650 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:25.532606 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:25.533607 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:08:25.541731 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:25.573794 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:25.696257 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:25.764180 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:25.769779 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:25.782425 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:25.789799 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:25.807822 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:25.869639 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:25.883033 17430 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:25.883263 17430 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:25.884011 17430 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:25.885555 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:25.886363 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:08:25.887596 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:25.887760 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:25.888849 17430 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:26.039286 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:26.049568 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:26.050424 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:26.067298 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:26.075994 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:26.088927 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:26.114903 17640 raft_consensus.cc:998] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.058s)
I20240501 14:08:26.115809 17640 consensus_queue.cc:237] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
W20240501 14:08:26.120199 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:08:26.138391 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:26.143376 28875 raft_consensus.cc:1261] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:26.148144 17736 consensus_queue.cc:1035] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:08:26.151543 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:26.159894 28875 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:26.158610 17728 raft_consensus.cc:2932] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
I20240501 14:08:26.165429 28604 catalog_manager.cc:5484] T 763b2e84afa64924ad5eeeeb3265a1d8 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 6 to 7, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } }
W20240501 14:08:26.188302 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:26.201898 16895 raft_consensus.cc:998] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.028s)
I20240501 14:08:26.202522 16895 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:26.205113 30926 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 14:08:26.205744 17751 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:08:26.208895 16895 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.209607 30926 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.218600 28599 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 10 to 11, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 2 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.233144 17751 raft_consensus.cc:998] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.069s)
I20240501 14:08:26.233987 17751 consensus_queue.cc:237] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:26.237068 29007 consensus_queue.cc:237] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:26.242210 30926 raft_consensus.cc:1261] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 14:08:26.243363 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:26.243891 17751 consensus_queue.cc:1035] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:08:26.244330 30926 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:08:26.245081 16895 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:08:26.247995 16895 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.248814 30926 raft_consensus.cc:2932] T e0bf08cbb6d347d783b1fa7762c802df P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.259785 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e0bf08cbb6d347d783b1fa7762c802df with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:26.263466 28599 catalog_manager.cc:5484] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 11 to 12, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New cstate: current_term: 2 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
W20240501 14:08:26.290747 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:08:26.291122 17756 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.295568 30925 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.304271 28599 catalog_manager.cc:5484] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.310598 29007 consensus_queue.cc:237] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
W20240501 14:08:26.313865 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:26.318424 30925 raft_consensus.cc:1261] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:08:26.319370 17756 consensus_queue.cc:1035] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:08:26.322836 17756 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.326774 30925 raft_consensus.cc:2932] T 12510f71b6a742ab8cc359243de8119b P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.334025 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 12510f71b6a742ab8cc359243de8119b with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:26.335995 28604 catalog_manager.cc:5484] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.439473 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:26.440631 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:26.470006 17608 raft_consensus.cc:998] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.404s)
I20240501 14:08:26.471323 17608 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:26.475030 30925 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:08:26.475399 16988 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:26.475582 16988 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:26.476708 16988 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:08:26.479676 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:08:26.480036 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:08:26.489596 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:26.491827 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:26.492621 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:26.492812 30862 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:26.497152 16988 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:26.520176 17608 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:08:26.528275 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 45384.9us
I20240501 14:08:26.558740 30926 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.562965 17608 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.566157 28600 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 8 to 9, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 2 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.568900 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:26.587683 28875 consensus_queue.cc:237] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:26.590471 30925 raft_consensus.cc:1261] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 14:08:26.591305 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:26.591317 17800 consensus_queue.cc:1035] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240501 14:08:26.595330 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:26.598989 17430 raft_consensus.cc:998] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.287s)
I20240501 14:08:26.599895 17430 consensus_queue.cc:237] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:26.607509 17800 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.611621 30925 raft_consensus.cc:2932] T de161b10c84f4bcc85f8cf7b199f99db P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.615453 30926 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:08:26.616319 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet de161b10c84f4bcc85f8cf7b199f99db with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:26.618178 28600 catalog_manager.cc:5484] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 9 to 10, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New cstate: current_term: 2 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.619201 17430 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:08:26.625201 17757 raft_consensus.cc:998] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.558s)
I20240501 14:08:26.625825 17757 consensus_queue.cc:237] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:26.641340 28745 raft_consensus.cc:1261] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:08:26.642033 17755 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:08:26.655479 17755 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.661955 28745 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.668567 28604 catalog_manager.cc:5484] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 5 to 6, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.671244 17430 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.673720 30925 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.676311 29007 consensus_queue.cc:237] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
W20240501 14:08:26.678740 28940 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Couldn't send request to peer cd3c04ebb6c84b15a1b3adc748737094. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9bb3eaf3f0e40f8902b9426a845d9b2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:26.679195 28745 raft_consensus.cc:1261] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:08:26.679800 17757 consensus_queue.cc:1035] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:08:26.680141 28604 catalog_manager.cc:5484] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.683211 17756 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:26.688810 28745 raft_consensus.cc:2932] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
W20240501 14:08:26.697189 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:08:26.697294 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e9bb3eaf3f0e40f8902b9426a845d9b2 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:26.698097 28604 catalog_manager.cc:5484] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 7, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:08:26.699182 29136 consensus_queue.cc:237] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:26.706864 30925 raft_consensus.cc:1261] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:08:26.707696 17430 consensus_queue.cc:1035] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:08:26.710844 17813 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
W20240501 14:08:26.714354 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f516f01e242d4b8789130201f748f3ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:26.715512 30926 raft_consensus.cc:2932] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.717262 17819 raft_consensus.cc:998] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.539s)
I20240501 14:08:26.718225 17819 consensus_queue.cc:237] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:26.722458 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f516f01e242d4b8789130201f748f3ea with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:26.723577 30925 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:08:26.724354 28604 catalog_manager.cc:5484] T f516f01e242d4b8789130201f748f3ea P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.724833 17813 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:08:26.728056 17813 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.728771 30925 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
W20240501 14:08:26.732506 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:26.734470 28600 catalog_manager.cc:5484] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:08:26.739940 29136 consensus_queue.cc:237] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
W20240501 14:08:26.742843 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:08:26.743324 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:26.743593 30925 raft_consensus.cc:1261] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:08:26.744200 17813 consensus_queue.cc:1035] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:08:26.747102 17813 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.747711 30925 raft_consensus.cc:2932] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.752035 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 86bfedcdabe44e7ab135674163935786 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:08:26.754673 28600 catalog_manager.cc:5484] T 86bfedcdabe44e7ab135674163935786 P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 8 to 9, VOTER 343a85ba3b514663af5320cd14a9e144 (127.16.36.129) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:08:26.801581 17836 ts_tablet_manager.cc:902] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:08:26.826002 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:26.845640 17836 tablet_copy_client.cc:308] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
W20240501 14:08:26.846372 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:08:26.847733 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet e0bf08cbb6d347d783b1fa7762c802df from peer 343a85ba3b514663af5320cd14a9e144 ({username='slave'} at 127.16.36.129:37221)
I20240501 14:08:26.847885 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet e0bf08cbb6d347d783b1fa7762c802df from peer 343a85ba3b514663af5320cd14a9e144 at {username='slave'} at 127.16.36.129:37221: session id = 343a85ba3b514663af5320cd14a9e144-e0bf08cbb6d347d783b1fa7762c802df
I20240501 14:08:26.853449 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:26.863515 29027 tablet_copy_source_session.cc:205] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:26.865330 17836 ts_tablet_manager.cc:1875] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240501 14:08:26.867354 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:08:26.872985 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:26.885020 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:26.890105 17836 ts_tablet_manager.cc:1888] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240501 14:08:26.892204 17836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bf08cbb6d347d783b1fa7762c802df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:26.893349 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:26.897039 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:26.897742 17836 tablet_copy_client.cc:783] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:26.907807 17836 tablet_copy_client.cc:647] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:26.912065 17851 ts_tablet_manager.cc:902] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:08:26.917115 17836 tablet_copy_client.cc:523] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:26.924995 17836 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
W20240501 14:08:26.935101 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:26.940112 17851 tablet_copy_client.cc:308] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:08:26.941009 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 12510f71b6a742ab8cc359243de8119b from peer 343a85ba3b514663af5320cd14a9e144 ({username='slave'} at 127.16.36.129:37221)
I20240501 14:08:26.941205 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 12510f71b6a742ab8cc359243de8119b from peer 343a85ba3b514663af5320cd14a9e144 at {username='slave'} at 127.16.36.129:37221: session id = 343a85ba3b514663af5320cd14a9e144-12510f71b6a742ab8cc359243de8119b
I20240501 14:08:26.945024 29027 tablet_copy_source_session.cc:205] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:26.947444 17851 ts_tablet_manager.cc:1875] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:08:26.956532 17851 ts_tablet_manager.cc:1888] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:08:26.956976 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12510f71b6a742ab8cc359243de8119b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:26.961589 17851 tablet_copy_client.cc:783] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:26.969347 17851 tablet_copy_client.cc:647] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:26.972102 17851 tablet_copy_client.cc:523] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:08:26.972340 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:08:26.976930 17851 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:08:27.013238 17851 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:27.013796 17851 tablet_bootstrap.cc:492] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Bootstrap complete.
I20240501 14:08:27.014132 17851 ts_tablet_manager.cc:1367] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.003s
I20240501 14:08:27.015161 17851 raft_consensus.cc:348] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.015398 17851 raft_consensus.cc:729] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:08:27.015690 17851 consensus_queue.cc:260] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.017156 17851 ts_tablet_manager.cc:1398] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:27.017882 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 343a85ba3b514663af5320cd14a9e144-12510f71b6a742ab8cc359243de8119b received from {username='slave'} at 127.16.36.129:37221
I20240501 14:08:27.018026 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 343a85ba3b514663af5320cd14a9e144-12510f71b6a742ab8cc359243de8119b on tablet 12510f71b6a742ab8cc359243de8119b with peer 343a85ba3b514663af5320cd14a9e144
I20240501 14:08:27.027843 17851 ts_tablet_manager.cc:902] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:27.036103 17836 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:27.036767 17836 tablet_bootstrap.cc:492] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Bootstrap complete.
I20240501 14:08:27.037194 17836 ts_tablet_manager.cc:1367] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.112s user 0.022s sys 0.013s
I20240501 14:08:27.038403 17836 raft_consensus.cc:348] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.039266 17836 raft_consensus.cc:729] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:08:27.039570 17836 consensus_queue.cc:260] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.040208 17836 ts_tablet_manager.cc:1398] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:27.042726 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 343a85ba3b514663af5320cd14a9e144-e0bf08cbb6d347d783b1fa7762c802df received from {username='slave'} at 127.16.36.129:37221
I20240501 14:08:27.042917 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 343a85ba3b514663af5320cd14a9e144-e0bf08cbb6d347d783b1fa7762c802df on tablet e0bf08cbb6d347d783b1fa7762c802df with peer 343a85ba3b514663af5320cd14a9e144
I20240501 14:08:27.053285 17851 tablet_copy_client.cc:308] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:08:27.054351 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet de161b10c84f4bcc85f8cf7b199f99db from peer 343a85ba3b514663af5320cd14a9e144 ({username='slave'} at 127.16.36.129:56137)
I20240501 14:08:27.054546 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet de161b10c84f4bcc85f8cf7b199f99db from peer 343a85ba3b514663af5320cd14a9e144 at {username='slave'} at 127.16.36.129:56137: session id = 343a85ba3b514663af5320cd14a9e144-de161b10c84f4bcc85f8cf7b199f99db
I20240501 14:08:27.057585 28896 tablet_copy_source_session.cc:205] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:27.058892 17851 ts_tablet_manager.cc:1875] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:08:27.065616 17851 ts_tablet_manager.cc:1888] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:08:27.065938 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de161b10c84f4bcc85f8cf7b199f99db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:27.069491 17851 tablet_copy_client.cc:783] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:27.076879 17851 tablet_copy_client.cc:647] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:27.078756 17851 tablet_copy_client.cc:523] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:27.081972 17851 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:08:27.099932 17851 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:27.100342 17851 tablet_bootstrap.cc:492] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Bootstrap complete.
I20240501 14:08:27.100615 17851 ts_tablet_manager.cc:1367] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.019s user 0.020s sys 0.000s
I20240501 14:08:27.101387 17851 raft_consensus.cc:348] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.101635 17851 raft_consensus.cc:729] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:08:27.101928 17851 consensus_queue.cc:260] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.102519 17851 ts_tablet_manager.cc:1398] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:08:27.103438 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 343a85ba3b514663af5320cd14a9e144-de161b10c84f4bcc85f8cf7b199f99db received from {username='slave'} at 127.16.36.129:56137
I20240501 14:08:27.103601 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 343a85ba3b514663af5320cd14a9e144-de161b10c84f4bcc85f8cf7b199f99db on tablet de161b10c84f4bcc85f8cf7b199f99db with peer 343a85ba3b514663af5320cd14a9e144
W20240501 14:08:27.117159 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:27.151091 17890 ts_tablet_manager.cc:902] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:08:27.160462 17890 tablet_copy_client.cc:308] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:08:27.161232 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet e9bb3eaf3f0e40f8902b9426a845d9b2 from peer cd3c04ebb6c84b15a1b3adc748737094 ({username='slave'} at 127.16.36.130:47511)
I20240501 14:08:27.161397 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet e9bb3eaf3f0e40f8902b9426a845d9b2 from peer cd3c04ebb6c84b15a1b3adc748737094 at {username='slave'} at 127.16.36.130:47511: session id = cd3c04ebb6c84b15a1b3adc748737094-e9bb3eaf3f0e40f8902b9426a845d9b2
I20240501 14:08:27.166469 29027 tablet_copy_source_session.cc:205] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:27.169679 17890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bb3eaf3f0e40f8902b9426a845d9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:27.181869 17890 tablet_copy_client.cc:783] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:27.186692 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:27.191606 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:27.195247 17890 tablet_copy_client.cc:647] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:27.199060 17890 tablet_copy_client.cc:523] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:08:27.198994 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet de161b10c84f4bcc85f8cf7b199f99db: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:27.205782 17890 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:08:27.218541 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(12510f71b6a742ab8cc359243de8119b): 4921 bytes on disk
I20240501 14:08:27.218998 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:27.220101 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(12510f71b6a742ab8cc359243de8119b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240501 14:08:27.221580 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(de161b10c84f4bcc85f8cf7b199f99db): 4581 bytes on disk
I20240501 14:08:27.222287 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(de161b10c84f4bcc85f8cf7b199f99db) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 14:08:27.226979 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(e0bf08cbb6d347d783b1fa7762c802df): 4581 bytes on disk
I20240501 14:08:27.227907 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(e0bf08cbb6d347d783b1fa7762c802df) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240501 14:08:27.228628 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:27.235224 17851 ts_tablet_manager.cc:902] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:08:27.238550 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:27.241016 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:27.250382 17836 ts_tablet_manager.cc:902] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: Initiating tablet copy from peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
W20240501 14:08:27.300216 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:27.300503 17851 tablet_copy_client.cc:308] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:08:27.301309 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet 86bfedcdabe44e7ab135674163935786 from peer 343a85ba3b514663af5320cd14a9e144 ({username='slave'} at 127.16.36.129:51037)
I20240501 14:08:27.301481 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet 86bfedcdabe44e7ab135674163935786 from peer 343a85ba3b514663af5320cd14a9e144 at {username='slave'} at 127.16.36.129:51037: session id = 343a85ba3b514663af5320cd14a9e144-86bfedcdabe44e7ab135674163935786
I20240501 14:08:27.305524 29158 tablet_copy_source_session.cc:205] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:27.307083 17851 ts_tablet_manager.cc:1875] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:08:27.327534 28745 raft_consensus.cc:1203] T e0bf08cbb6d347d783b1fa7762c802df P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 14:08:27.335563 17836 tablet_copy_client.cc:308] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.132:42827
I20240501 14:08:27.336400 29158 tablet_copy_service.cc:140] P 3a3972b170c54344a4ba61f176f837ba: Received BeginTabletCopySession request for tablet f516f01e242d4b8789130201f748f3ea from peer 343a85ba3b514663af5320cd14a9e144 ({username='slave'} at 127.16.36.129:51037)
I20240501 14:08:27.336452 17851 ts_tablet_manager.cc:1888] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:08:27.336582 29158 tablet_copy_service.cc:161] P 3a3972b170c54344a4ba61f176f837ba: Beginning new tablet copy session on tablet f516f01e242d4b8789130201f748f3ea from peer 343a85ba3b514663af5320cd14a9e144 at {username='slave'} at 127.16.36.129:51037: session id = 343a85ba3b514663af5320cd14a9e144-f516f01e242d4b8789130201f748f3ea
I20240501 14:08:27.336855 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bfedcdabe44e7ab135674163935786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:27.340399 29158 tablet_copy_source_session.cc:205] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:08:27.341940 17836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f516f01e242d4b8789130201f748f3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:27.345265 17851 tablet_copy_client.cc:783] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:27.349922 17836 tablet_copy_client.cc:783] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 5 data blocks...
I20240501 14:08:27.353504 17851 tablet_copy_client.cc:647] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:27.362157 17851 tablet_copy_client.cc:523] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:27.374332 17836 tablet_copy_client.cc:647] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:27.377612 17836 tablet_copy_client.cc:523] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:27.375449 17851 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:08:27.382325 17836 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: Bootstrap starting.
I20240501 14:08:27.393913 17890 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:27.394343 17890 tablet_bootstrap.cc:492] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap complete.
I20240501 14:08:27.396014 17890 ts_tablet_manager.cc:1367] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.190s user 0.077s sys 0.046s
I20240501 14:08:27.397006 17890 raft_consensus.cc:348] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:27.397241 17890 raft_consensus.cc:729] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:08:27.398706 17890 consensus_queue.cc:260] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:27.401402 17890 ts_tablet_manager.cc:1398] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:08:27.403245 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session cd3c04ebb6c84b15a1b3adc748737094-e9bb3eaf3f0e40f8902b9426a845d9b2 received from {username='slave'} at 127.16.36.130:47511
I20240501 14:08:27.403438 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session cd3c04ebb6c84b15a1b3adc748737094-e9bb3eaf3f0e40f8902b9426a845d9b2 on tablet e9bb3eaf3f0e40f8902b9426a845d9b2 with peer cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:27.415879 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(e9bb3eaf3f0e40f8902b9426a845d9b2): 4581 bytes on disk
I20240501 14:08:27.416679 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(e9bb3eaf3f0e40f8902b9426a845d9b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 14:08:27.428084 28745 raft_consensus.cc:1203] T 12510f71b6a742ab8cc359243de8119b P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:08:27.472669 17851 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:27.473214 17851 tablet_bootstrap.cc:492] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Bootstrap complete.
I20240501 14:08:27.473555 17851 ts_tablet_manager.cc:1367] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.098s user 0.029s sys 0.005s
I20240501 14:08:27.474443 17851 raft_consensus.cc:348] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.474733 17851 raft_consensus.cc:729] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:08:27.475029 17851 consensus_queue.cc:260] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
W20240501 14:08:27.475623 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:27.475991 17851 ts_tablet_manager.cc:1398] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 14:08:27.476984 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 343a85ba3b514663af5320cd14a9e144-86bfedcdabe44e7ab135674163935786 received from {username='slave'} at 127.16.36.129:51037
I20240501 14:08:27.477118 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 343a85ba3b514663af5320cd14a9e144-86bfedcdabe44e7ab135674163935786 on tablet 86bfedcdabe44e7ab135674163935786 with peer 343a85ba3b514663af5320cd14a9e144
I20240501 14:08:27.483047 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(86bfedcdabe44e7ab135674163935786): 4834 bytes on disk
I20240501 14:08:27.483829 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(86bfedcdabe44e7ab135674163935786) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:08:27.484304 17836 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:27.484761 17836 tablet_bootstrap.cc:492] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: Bootstrap complete.
I20240501 14:08:27.484992 17836 ts_tablet_manager.cc:1367] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: Time spent bootstrapping tablet: real 0.103s user 0.041s sys 0.038s
I20240501 14:08:27.485597 17836 raft_consensus.cc:348] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.485839 17836 raft_consensus.cc:729] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a85ba3b514663af5320cd14a9e144, State: Initialized, Role: FOLLOWER
I20240501 14:08:27.486083 17836 consensus_queue.cc:260] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:08:27.486593 17836 ts_tablet_manager.cc:1398] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:08:27.487301 29158 tablet_copy_service.cc:342] P 3a3972b170c54344a4ba61f176f837ba: Request end of tablet copy session 343a85ba3b514663af5320cd14a9e144-f516f01e242d4b8789130201f748f3ea received from {username='slave'} at 127.16.36.129:51037
I20240501 14:08:27.487452 29158 tablet_copy_service.cc:433] P 3a3972b170c54344a4ba61f176f837ba: ending tablet copy session 343a85ba3b514663af5320cd14a9e144-f516f01e242d4b8789130201f748f3ea on tablet f516f01e242d4b8789130201f748f3ea with peer 343a85ba3b514663af5320cd14a9e144
I20240501 14:08:27.525038 28875 raft_consensus.cc:1203] T e9bb3eaf3f0e40f8902b9426a845d9b2 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:08:27.577463 28745 raft_consensus.cc:1203] T de161b10c84f4bcc85f8cf7b199f99db P 343a85ba3b514663af5320cd14a9e144 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 14:08:27.625617 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:27.638923 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:27.651249 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:27.653013 28743 raft_consensus.cc:1203] T f516f01e242d4b8789130201f748f3ea P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:08:27.658811 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:27.659101 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:27.659801 28743 raft_consensus.cc:1203] T 86bfedcdabe44e7ab135674163935786 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240501 14:08:27.721154 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:27.734306 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:08:27.736842 28791 maintenance_manager.cc:419] P 343a85ba3b514663af5320cd14a9e144: Scheduling UndoDeltaBlockGCOp(f516f01e242d4b8789130201f748f3ea): 4581 bytes on disk
I20240501 14:08:27.737541 28682 maintenance_manager.cc:643] P 343a85ba3b514663af5320cd14a9e144: UndoDeltaBlockGCOp(f516f01e242d4b8789130201f748f3ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 14:08:27.886368 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:27.888053 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:08:27.903934 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:27.924769 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:27.926090 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:27.962708 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:08:27.964272 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:28.145063 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:28.164013 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:28.165939 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:28.167012 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:28.177055 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:28.190541 16988 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:28.190816 16988 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:28.191591 16988 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:08:28.192711 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:08:28.192977 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:08:28.194700 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:28.203846 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:28.224916 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:28.225119 30862 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:28.225432 16988 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:28.329921 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 14:08:28.401225 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:28.407625 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:28.414705 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:28.428701 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:28.461117 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:28.465780 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:28.481726 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:08:28.596679 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:08:28.601691 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:28.695163 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:28.707032 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:28.707847 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:28.723824 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:28.744303 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:28.778951 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:08:28.781998 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:28.784982 17813 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:28.785200 17813 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:28.785955 17813 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:08:28.787480 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:08:28.789620 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:28.790015 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:28.790138 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:28.790807 17813 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:28.997359 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:29.010156 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.015727 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.024662 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:29.057698 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:29.075476 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240501 14:08:29.085074 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:08:29.111836 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:29.224699 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:08:29.242863 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:08:29.256493 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:08:29.260501 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:08:29.261495 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.270447 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.282347 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:29.284629 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:29.287886 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:08:29.296491 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:29.303956 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:29.358748 16988 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:08:29.358975 16988 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:29.359930 16988 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:08:29.362983 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:29.363209 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:29.363302 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:29.363617 16988 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:29.516099 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.518616 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.535686 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:29.540838 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:29.559787 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:29.602771 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:08:29.604055 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:08:29.634447 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:29.807163 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.808907 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:29.812932 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:29.818370 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:29.829645 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:29.909871 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:08:30.042114 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:08:30.058797 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:30.059010 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:30.071283 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:30.091881 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:30.103879 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:30.108479 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:30.294374 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:08:30.341996 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:30.343978 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:30.353758 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:30.358798 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:30.362802 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:30.520186 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:30.587822 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:30.588421 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:30.596051 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:08:30.605799 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:30.613803 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:30.632241 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:30.646042 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:30.805519 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:08:30.825080 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:08:30.850613 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:30.852352 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:30.866582 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:30.870990 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:30.887264 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:31.076202 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.080405 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:31.086179 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.092952 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:31.098975 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:31.178823 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:31.294147 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:08:31.297350 28593 maintenance_manager.cc:419] P 1bdce3de4d694d6c98f909a5b3c450ce: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7636 bytes on disk
I20240501 14:08:31.315567 28592 maintenance_manager.cc:643] P 1bdce3de4d694d6c98f909a5b3c450ce: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.003s sys 0.008s Metrics: {"thread_start_us":7732,"threads_started":2,"wal-append.queue_time_us":3483}
I20240501 14:08:31.336002 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.348192 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:31.371006 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.381956 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:31.387401 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:31.392371 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:08:31.410223 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:08:31.447687 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240501 14:08:31.520087 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:08:31.549355 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:08:31.553385 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:08:31.608397 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.622324 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.630688 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:31.640683 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:31.652196 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:31.782994 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:08:31.818773 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:31.851106 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.861604 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:31.869374 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:31.877938 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:31.902477 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:31.937670 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 14:08:32.063593 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:08:32.106046 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:08:32.149063 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.162940 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.170691 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:32.172801 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:32.172830 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:32.201835 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:32.263628 16988 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:32.263846 16988 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:32.264814 16988 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:08:32.266353 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:08:32.266716 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:08:32.269582 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:32.284111 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:32.284310 30862 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:32.284754 16988 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:32.418489 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:08:32.442032 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.452035 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:32.456570 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.467463 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:32.478879 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:32.495543 17813 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:32.495790 17813 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:32.496600 17813 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:32.497253 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:32.498224 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:08:32.498778 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:32.498926 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:32.499637 17813 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:32.607335 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:08:32.702236 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.709124 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:32.710812 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:32.713968 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:32.719167 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.729967 18722 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:32.730198 18722 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:32.730952 18722 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:08:32.733914 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:32.734247 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:32.734365 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:32.734937 18722 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:32.815913 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:32.827628 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:32.940594 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.942122 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:32.942914 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:32.960441 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:32.966662 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:33.014784 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 66740.2us
I20240501 14:08:33.174796 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:33.177480 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:33.183193 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:33.212920 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:33.227708 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:33.231626 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:08:33.304935 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:33.399277 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:08:33.401593 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:08:33.410610 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:33.468890 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:33.471962 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:33.480314 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:33.481346 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:33.484409 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:33.741057 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:33.741429 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:33.748344 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:33.752846 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:33.770926 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:33.848780 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240501 14:08:33.885668 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:08:33.898315 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:33.925727 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:08:34.031961 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:34.033012 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:34.038928 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:34.039618 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:08:34.049665 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:08:34.050166 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 14:08:34.051265 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:34.062909 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:34.143735 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:08:34.180985 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:34.249068 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:34.250280 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:34.262473 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:34.262594 16988 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:34.262852 16988 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:34.263935 16988 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:08:34.265419 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:08:34.265659 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:08:34.267220 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:34.267525 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:34.267642 30862 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:34.267922 16988 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:34.270529 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:34.275568 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:34.324713 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:08:34.352546 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:08:34.356523 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 14:08:34.477615 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:34.536972 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:34.557036 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:34.559855 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:34.563203 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:34.582605 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:34.612743 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:08:34.619513 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:08:34.633296 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:34.647122 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:34.798499 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:34.806483 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:34.821004 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:34.827936 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:34.855134 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:34.857002 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:35.043159 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:35.053264 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:35.053898 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:35.055699 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:35.059792 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:35.074291 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:35.153510 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:08:35.154896 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:35.274340 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:35.286033 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:35.295523 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:35.313872 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:35.319226 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:35.329104 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:35.545315 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:35.555840 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:35.564157 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:35.564198 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:35.573233 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:35.595360 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:35.708014 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:35.810477 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:35.814877 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:35.829457 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:35.829500 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:35.841094 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:35.858716 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:35.860606 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:36.023339 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:36.088395 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.094930 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:36.097432 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:36.100062 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.122327 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:36.183688 18722 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:36.183873 18722 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:36.184705 18722 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:36.185375 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:36.186120 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:08:36.186591 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:36.186761 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:36.187359 18722 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:36.242734 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:36.325088 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.341542 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:36.342746 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.344846 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:36.358155 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:36.453198 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:08:36.518534 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:08:36.611202 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:36.635293 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.635800 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.637182 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:36.648109 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:36.665717 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:36.671509 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:36.709378 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:36.716812 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:08:36.741232 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:08:36.830766 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 14:08:36.832100 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:36.884835 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.887691 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:36.893867 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:36.896494 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:36.917315 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:36.937794 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 14:08:37.081514 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:37.118381 18722 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:37.118587 18722 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:37.119361 18722 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:08:37.122426 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:37.123935 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:37.133433 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:37.151238 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:37.154599 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:37.176165 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:37.176426 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:37.176750 18722 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:37.189261 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:37.217515 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:08:37.313051 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:37.430439 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:37.432832 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:37.438310 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:37.443004 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:37.461629 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:37.462878 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:37.619601 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:37.691614 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:37.692649 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:37.702122 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:37.718364 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:37.725466 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:37.794917 19485 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:08:37.795143 19485 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:08:37.796167 19485 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:08:37.798383 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:37.799362 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:37.799523 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:08:37.799881 19485 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:37.885648 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:08:37.929476 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:37.931064 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:37.935700 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:37.951035 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:37.952593 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:37.957998 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:38.165215 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:08:38.186013 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:08:38.190546 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:38.196617 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:38.201053 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:38.205286 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:38.228227 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:38.231988 19442 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:38.232167 19442 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:38.233152 19442 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:08:38.235025 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:38.235275 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:08:38.235391 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:38.235677 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:38.235833 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:38.236075 19442 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:38.408274 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:08:38.409420 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:38.463492 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:38.470216 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:38.472312 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:38.488740 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:38.490350 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:38.491317 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:38.714617 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:38.723814 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:38.724864 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:38.742300 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:38.743508 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:38.792676 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:08:38.970288 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 34 similar messages]
I20240501 14:08:38.997100 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.013594 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.036851 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:39.050325 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 14:08:39.050855 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:39.060166 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:39.062934 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:39.086287 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:08:39.178577 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:08:39.187233 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:08:39.274310 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.284998 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.314119 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:39.324868 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:39.331063 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:39.503115 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:08:39.519533 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:08:39.549153 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.550333 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.564380 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:39.566913 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:39.580349 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:39.587922 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:39.592303 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:39.676373 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:08:39.677382 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 14:08:39.679986 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:08:39.690717 19485 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:08:39.690966 19485 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:08:39.691720 19485 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:08:39.693255 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:39.693747 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:08:39.694366 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:39.694463 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:08:39.694756 19485 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:39.780962 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.781714 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:39.788014 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:39.791731 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:39.815331 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:39.918951 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:08:40.043110 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:40.048642 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:40.059131 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:40.066732 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:40.087606 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:40.093281 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:08:40.099859 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:40.288307 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:40.291785 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:40.317945 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:40.322803 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:40.345755 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:40.444818 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:40.550336 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:40.550338 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:40.559357 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:40.563279 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:40.580901 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:40.586808 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:08:40.693977 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:40.811792 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:40.838232 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:40.839913 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:40.840862 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:40.868729 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:40.879446 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:40.916738 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:08:40.917019 19442 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:40.917178 19442 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:40.917989 19442 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:08:40.919351 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:40.934726 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:40.935009 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:08:40.936053 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:40.936216 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:40.936555 19442 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:41.065272 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.066164 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:41.079478 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:41.082089 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.092325 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:41.229394 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:41.252921 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:08:41.359529 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.361407 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.380151 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:41.390722 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:41.396968 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:41.604903 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.610831 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.611871 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:41.629082 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:41.639989 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:41.705479 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:41.717126 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:41.735149 20134 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:41.735361 20134 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:41.736132 20134 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:08:41.736793 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:41.737475 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:08:41.738008 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:41.738129 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:41.738405 20134 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:41.775905 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:08:41.776633 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:08:41.787566 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:08:41.834144 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 14:08:41.835111 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.848044 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:41.848665 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:41.866874 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:41.869886 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:41.870890 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 14:08:42.070434 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:42.082944 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:42.089282 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:42.096905 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:42.101508 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:42.184365 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:08:42.226974 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 14:08:42.247684 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:08:42.248821 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:08:42.275985 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:42.321681 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:42.323127 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:42.328971 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:42.355163 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:42.359931 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:42.457844 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:42.563009 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:42.571520 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:42.593676 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:42.607324 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:42.621207 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:42.631117 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:42.843649 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:42.844838 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:42.878175 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:42.889344 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:08:42.896031 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 16 similar messages]
I20240501 14:08:42.904371 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:42.904791 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:42.908977 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:42.932751 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:43.109841 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:43.141832 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:43.153617 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:43.154256 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:43.170779 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:08:43.192270 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:43.192508 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:43.225015 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:08:43.421638 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:43.425460 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:43.427948 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:43.441041 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:43.452581 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:43.453414 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:43.466471 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:08:43.607605 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:08:43.685568 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:43.688721 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:43.709983 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:43.712957 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:43.744433 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:43.937723 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:43.941576 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:43.943987 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:43.952710 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:43.971829 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:43.989039 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:44.039979 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:08:44.062530 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:44.186236 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:44.192399 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:44.193295 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:44.209393 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:44.211290 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:44.265877 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:08:44.279316 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:08:44.335179 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 14:08:44.385581 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 14:08:44.435161 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:44.442605 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:44.452754 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:44.455603 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:44.462077 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:44.492928 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:08:44.525836 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:44.711777 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:44.721230 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:44.723891 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:44.729487 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:44.733585 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 14:08:44.739972 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:44.844137 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:08:44.873674 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:44.937144 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:08:44.941946 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:44.942368 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:44.962246 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:44.970255 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 14:08:44.972110 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:44.973974 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:44.982672 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:45.054384 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:08:45.054543 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:08:45.229535 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:45.270720 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:45.293407 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:45.296985 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:45.314260 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:45.525738 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:45.530328 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:45.535976 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:45.536521 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:45.549484 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:08:45.555161 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:45.555976 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:45.563721 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:08:45.762079 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:45.777477 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:45.778544 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:45.787570 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:08:45.800544 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:45.825915 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:45.968348 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:08:45.978730 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:46.021102 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:46.030354 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:46.030458 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:08:46.031299 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:46.049407 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:46.077893 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:46.262243 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:46.270009 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:46.283865 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:46.284497 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:46.308283 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:46.495924 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:46.526878 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:08:46.553383 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:46.558938 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:46.569665 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:46.570830 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:46.597290 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:46.637136 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:08:46.728803 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:08:46.730221 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:08:46.825248 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:46.833978 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240501 14:08:46.845345 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:46.846971 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:08:46.863669 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:46.880316 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:46.884095 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:47.004500 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:47.097815 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:47.109663 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:47.120723 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:47.131204 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:47.156316 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:47.184844 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:08:47.364429 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:47.369495 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:47.380887 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:47.385092 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:47.407538 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:47.430476 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:08:47.486339 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 14:08:47.503548 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:08:47.510839 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:47.561967 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:08:47.624711 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:47.626677 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:47.627379 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:47.638080 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:47.658406 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:47.682937 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:08:47.822957 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:08:47.872215 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:47.874996 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:47.878301 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:47.892894 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:47.903065 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:47.965312 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:48.007596 19126 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:08:48.007840 19126 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:08:48.008617 19126 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:08:48.010186 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:48.010380 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:48.010474 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:48.010756 19126 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:48.081878 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:48.115772 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.119503 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.129830 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:48.136471 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:48.147296 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:48.362493 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:08:48.392524 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:48.403473 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.415776 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.424499 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:48.442000 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:48.452133 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:08:48.467278 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:48.477368 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:48.662907 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.663501 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:48.664400 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.681241 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:48.692970 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:48.741863 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:08:48.888754 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.902900 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:48.916791 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:48.925032 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:48.926188 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:08:48.937937 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:48.951421 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 14:08:49.012596 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:08:49.099256 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:49.123008 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:49.125011 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:49.140584 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:49.143139 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:49.152215 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:49.156714 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:08:49.206825 28811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 54008us
W20240501 14:08:49.229007 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:08:49.230043 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:49.404318 21326 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:49.404600 21326 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:08:49.405597 21326 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:08:49.408502 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:49.408795 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:49.408844 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:49.408932 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:49.409471 21326 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:49.420360 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:49.429569 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 14:08:49.436261 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:49.447747 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:49.448804 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:49.486524 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:49.638374 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:49.665410 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:49.668728 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:49.688028 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:49.701895 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:49.701952 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:49.817850 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:08:49.895799 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:08:49.899315 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 14:08:49.903077 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:49.966425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:49.989624 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:49.995434 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:08:49.995867 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:49.996697 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:50.012838 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:50.036780 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:50.050156 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:50.219828 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:50.222815 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:50.229550 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:50.241503 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:50.244038 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:50.355497 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:50.481603 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:50.485494 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:50.494156 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:50.501672 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:50.502177 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:50.591239 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:50.672376 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:50.724301 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:50.728662 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:50.730062 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:50.735306 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:50.759703 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:50.829531 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:08:50.900585 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:50.985746 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.001545 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.008244 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:51.018991 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:51.019618 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:51.100358 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:08:51.187390 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 14:08:51.193222 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:08:51.193759 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fb04ec77da194832aa63d69aba307de3: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:51.196964 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:51.227810 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.240723 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:51.264786 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.266306 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:51.266752 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:51.503504 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.504289 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.504629 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:51.522300 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:51.538091 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:51.559718 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 14:08:51.648494 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:08:51.674933 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 14:08:51.694753 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:51.709960 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:51.766800 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.776087 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:51.783517 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:51.787580 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:51.806250 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:51.980067 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:52.011194 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:52.034994 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:52.037806 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:52.039727 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:52.047456 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:52.239202 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:08:52.267413 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:52.274653 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:52.283951 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:52.284600 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:52.286814 21326 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:52.286994 21326 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:08:52.287887 21326 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:08:52.289141 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:08:52.289397 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:08:52.290119 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:52.295137 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:52.295300 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:08:52.295578 21326 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:52.321820 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:52.323561 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:52.348490 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:52.411901 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:08:52.442313 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:08:52.501283 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240501 14:08:52.537215 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:52.550175 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:52.550191 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:52.562479 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:52.566387 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:52.714782 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:52.775076 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:52.778510 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:52.793408 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:52.804273 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:52.816900 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:52.863186 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:08:52.870668 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 14:08:52.886128 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:52.937170 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:08:53.056633 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:53.058945 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:53.066236 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:53.076046 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:53.088480 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:53.143543 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:08:53.268190 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:08:53.284714 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:53.296561 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:53.297439 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:53.341126 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:53.348825 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:53.390736 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:08:53.465658 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:53.578828 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:53.594584 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:53.629802 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:53.631881 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:53.640419 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:53.806887 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:08:53.852416 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:53.863340 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:53.865227 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:53.867849 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:53.883428 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:53.900019 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:53.919301 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:53.971444 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:08:54.022017 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:08:54.029776 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:08:54.077719 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:08:54.104447 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:54.114358 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:54.118537 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:54.120190 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:54.123823 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:54.293617 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:54.384239 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:54.385283 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:54.386443 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:54.394290 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:54.411545 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:54.450529 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240501 14:08:54.546800 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:08:54.563597 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:08:54.634851 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:54.640415 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:54.664944 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:54.672775 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:54.703686 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:54.840677 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:54.893131 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:54.902529 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:54.903275 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:54.904688 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:54.917845 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:54.921243 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:08:54.931344 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:08:54.991446 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 14:08:55.085453 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:08:55.112695 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:08:55.120493 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.130769 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:55.142900 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:55.145031 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.158197 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:55.222220 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:55.300563 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:08:55.367552 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:55.414384 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.421407 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.448558 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:55.490531 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:55.492063 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:55.495661 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:08:55.618197 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:08:55.665179 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:55.708204 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.712513 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:55.713485 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:55.732146 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.738359 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:55.770068 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:55.883576 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 14:08:55.957168 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.968259 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:55.971668 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:55.976183 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:55.980322 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:56.199491 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.203038 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.215452 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:56.221793 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:56.224583 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:56.296098 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:56.419116 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:08:56.465818 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:56.467533 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.482661 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:56.486876 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.490314 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:56.512992 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:08:56.577039 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 14:08:56.615211 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:56.716985 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.723248 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:56.735436 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.744153 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:56.762894 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:56.797005 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:56.876331 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240501 14:08:56.969614 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.975595 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:56.975973 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:56.987185 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:57.015630 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:57.029518 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:57.105690 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:08:57.214116 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:57.234619 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:57.239607 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:57.253897 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:57.260298 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:57.261366 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:57.283035 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:57.290737 21992 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:08:57.290954 21992 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:08:57.291865 21992 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:08:57.295723 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:08:57.322670 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 14:08:57.338105 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:08:57.338284 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:08:57.338582 21992 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:57.502903 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:57.512692 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:57.515060 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:57.533710 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:57.539348 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:57.649832 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:08:57.753999 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:57.757957 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:08:57.766954 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:57.799305 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:57.819835 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:57.820753 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:57.825281 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:57.826112 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:57.868067 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:08:57.875392 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:58.044800 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.058176 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:58.064260 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:58.077920 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:08:58.078344 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:58.084437 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:58.141275 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:08:58.270257 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 14:08:58.281338 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.289726 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:58.303512 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:58.306605 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:58.311590 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:58.339258 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:58.511716 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.517822 22426 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:08:58.518080 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.518034 22426 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:08:58.534783 22426 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
W20240501 14:08:58.536520 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:58.538873 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:58.539224 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:08:58.541172 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:08:58.541350 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:08:58.541831 22426 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:08:58.543330 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:58.548347 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:58.789036 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:08:58.804531 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.809944 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:58.827529 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:58.828867 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:58.843266 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:08:58.890712 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:08:59.033195 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 14:08:59.071019 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:59.072194 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.074896 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.075901 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:59.082904 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:59.149024 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:08:59.164443 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:08:59.255961 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:59.302728 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:08:59.308158 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:08:59.334350 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:08:59.346521 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:08:59.348851 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.358246 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:59.366290 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:08:59.393234 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:08:59.440953 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:59.617502 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.624194 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.630570 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:59.642328 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:08:59.666002 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:08:59.672171 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:08:59.724402 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 14:08:59.726919 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:08:59.728401 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:08:59.735028 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:08:59.869647 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:08:59.900247 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:08:59.916791 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.920841 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:08:59.931270 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:08:59.959061 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:08:59.972138 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:00.002797 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 14:09:00.165086 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:09:00.174072 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:00.185253 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:00.199069 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:00.205891 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 14:09:00.208951 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:00.215209 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:00.220844 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:09:00.248960 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:09:00.399942 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:09:00.430322 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:00.434327 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:00.434460 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:00.450918 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:00.454710 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:00.454916 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:00.476931 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:09:00.699848 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:00.700965 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:00.719516 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:00.720017 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:00.725164 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:00.938697 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:00.940447 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:00.948786 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:00.955353 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:00.973804 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 14:09:00.978659 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:00.979409 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:01.168046 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:01.184725 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:01.185688 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:01.188966 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:01.203409 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:01.405473 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:01.418138 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:01.444813 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:01.458680 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:01.467358 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:01.499272 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:09:01.504542 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:01.686026 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:01.702371 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:01.704020 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:09:01.705536 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:01.721593 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240501 14:09:01.735056 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:01.747140 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:01.766005 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:09:01.903393 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:09:01.906271 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:01.950392 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:01.957752 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:01.974375 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:01.987406 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:02.007800 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:02.104694 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:09:02.159724 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:09:02.159798 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240501 14:09:02.225201 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:02.233675 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:02.234062 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:02.236555 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:09:02.236698 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:02.255764 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:02.344465 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:02.353552 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 14:09:02.448700 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:09:02.504915 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:02.523816 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:02.525096 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:02.528741 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:02.547973 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:02.606050 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:02.782474 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:02.786901 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:02.787966 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:02.807298 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:02.810979 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:02.822959 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:02.856124 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:02.882411 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:02.950820 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:09:03.052316 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.068177 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:03.079751 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.080281 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:03.092571 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:03.125782 21125 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:03.126024 21125 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:03.127247 21125 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:09:03.128154 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:09:03.130086 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:03.134124 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:09:03.134378 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:09:03.146252 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:03.146533 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:03.146870 21125 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:03.231279 21125 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:09:03.231496 21125 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:09:03.232331 21125 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:09:03.233742 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:09:03.233930 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:03.234009 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:03.234254 21125 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:03.305225 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.309827 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.326668 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:03.329953 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:03.348685 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:03.366556 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:03.562186 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:03.564574 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.578807 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:03.585994 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.598505 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:03.666323 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:09:03.726554 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:03.819687 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.824038 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:03.846897 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:03.861585 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:03.878960 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:03.981753 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:09:04.118121 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:09:04.119791 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:04.120445 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240501 14:09:04.123201 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:04.147253 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 14:09:04.148062 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:04.154853 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:09:04.176457 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:04.219352 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:04.253185 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:04.431339 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:04.436064 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:04.438338 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:04.446102 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:04.448175 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:04.467892 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:04.546268 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:09:04.663707 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:04.685505 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:04.686836 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:04.690657 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:04.697907 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:04.734308 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:09:04.772419 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:04.779204 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:04.861193 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:04.869307 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240501 14:09:04.884585 23501 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:09:04.885118 23501 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:09:04.885965 23501 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:09:04.887400 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:09:04.887895 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:09:04.887995 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:09:04.888211 23501 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:04.912647 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:04.929155 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:04.930521 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:04.938148 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:04.943985 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:09:04.956517 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:04.981220 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 14:09:05.152439 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:09:05.166759 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:05.173635 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:05.174335 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:05.177902 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 14:09:05.186580 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:05.193722 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:05.329417 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:05.331152 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:09:05.340848 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:09:05.420346 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:05.423171 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:05.427223 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:05.434976 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:05.449947 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:05.451464 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:05.643059 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:05.643720 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:05.663892 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:05.675262 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:05.676174 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:05.677301 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:09:05.906018 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:05.910601 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:05.911536 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:05.920827 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:05.932705 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:05.956720 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:06.069482 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:09:06.175521 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:06.182035 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:06.192799 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:06.198101 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:06.199204 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.490573 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.500898 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:06.507241 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:06.510493 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:06.511776 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:06.533471 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:06.570726 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:06.579695 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:09:06.747298 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:09:06.780637 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:06.784802 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:06.787097 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:06.790714 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:06.815171 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:06.834023 23375 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:06.834209 23375 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:09:06.834913 23375 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:09:06.837961 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:09:06.838282 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:06.838378 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:06.838919 23375 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:06.858106 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:06.972427 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:09:07.021728 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.024273 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.051129 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:07.051867 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.075955 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:07.136734 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:07.153154 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 14:09:07.214421 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:09:07.260376 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.270859 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:07.271714 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.299342 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.306027 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:07.326231 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:09:07.339395 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:09:07.382555 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240501 14:09:07.517853 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.531064 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:07.531632 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.542032 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:07.542575 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:07.563721 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:09:07.633148 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:07.779779 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.792717 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:07.792703 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:07.811486 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:07.843611 23375 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:07.843844 23375 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:09:07.844614 23375 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:09:07.847757 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:09:07.848462 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:09:07.852766 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:07.852923 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:09:07.853571 23375 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:07.893627 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:07.959345 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:07.992421 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:08.032287 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:08.049301 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.050305 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.050807 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:08.054183 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.058122 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:08.125375 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:09:08.210906 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:08.217522 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240501 14:09:08.360321 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.367038 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.408087 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:08.415858 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:08.416345 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:08.541280 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:09:08.652949 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:08.658754 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.664727 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.673128 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:08.674163 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:08.680905 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:08.916271 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.921777 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:08.922996 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:08.938030 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:08.946138 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:09.004487 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:09:09.021410 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:09:09.030645 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:09:09.085656 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 14:09:09.163148 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:09.167635 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:09.170876 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:09.185418 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:09.196336 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:09.222009 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:09.241045 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:09.439779 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:09.441692 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:09:09.444983 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:09.452317 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:09:09.453917 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:09.468528 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.472537 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:09.489732 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:09.687402 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:09.695269 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:09.696496 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:09.708660 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:09.716913 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 14:09:09.753341 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:09.769198 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240501 14:09:09.788518 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:09.790616 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:09:09.933523 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:09.944824 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:09.949210 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:09.960239 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:09.964113 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:09.984954 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:10.154793 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:09:10.183470 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 14:09:10.205577 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:10.221217 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:10.230075 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:10.235991 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:10.243808 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:10.258128 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:09:10.271536 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:10.285198 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:09:10.479874 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:10.482364 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:10.498522 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.504606 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:10.508777 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:10.556666 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:10.651402 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:09:10.715103 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:10.716892 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:10.718456 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:10.737946 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:09:10.745270 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.751796 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:10.814530 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:10.981209 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:10.985634 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:10.993700 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:10.994079 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:11.000869 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.213793 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:11.222507 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:11.229746 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.247587 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:11.249997 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:11.268941 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:09:11.398849 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:11.464470 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:11.469944 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:11.471390 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:11.502959 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.524098 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:11.554737 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:09:11.654779 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:11.681440 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 14:09:11.770267 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:11.770901 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:11.772989 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:11.779630 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:11.781363 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:11.795153 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:11.807065 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:09:11.809374 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:09:11.998756 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.023945 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.033890 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:12.055404 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.077412 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:12.207877 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 14:09:12.217772 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:09:12.253358 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:09:12.254303 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.267351 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:12.269253 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.294301 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.298697 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:12.313992 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:12.318722 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240501 14:09:12.319957 24449 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:12.320133 24449 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:09:12.320870 24449 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:09:12.325604 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:12.328068 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:12.328197 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:12.330870 24449 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:12.486363 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:12.517313 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:12.522724 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.522886 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.535025 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:12.539594 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:12.653121 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:09:12.758054 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:12.763525 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.770773 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:12.772290 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:12.775667 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:12.775840 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:12.779685 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:12.871464 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:09:12.994918 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.009810 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.011003 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.034438 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:13.041858 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:13.067318 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:09:13.187479 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:09:13.255282 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.262887 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.275130 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:13.277225 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.278352 24939 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:13.278522 24939 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:13.286679 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:13.287032 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:13.287086 24939 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:09:13.288976 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:13.289208 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:09:13.289448 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:09:13.290061 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:13.290179 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:13.290447 24939 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:13.451951 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:09:13.542279 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:13.547936 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.549968 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.568567 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.595080 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:13.759541 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.769270 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:13.773824 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:13.779358 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.784276 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:13.838344 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:13.838495 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:09:13.906029 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:09:13.922899 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:09:13.995425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:13.998842 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.011658 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:14.041652 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:14.049893 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.067497 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:14.102319 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:09:14.149960 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:09:14.233732 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.242003 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:14.253463 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.262084 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:14.271531 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:14.278159 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:14.455415 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:14.494916 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.495046 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.514917 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:14.525972 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.537703 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:14.753492 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.756731 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:14.757712 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 14:09:14.758808 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:14.773419 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:14.791494 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:14.818917 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:09:14.826807 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:09:14.830802 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 14:09:14.854169 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:14.883888 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:14.990777 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:14.995195 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:15.011225 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.013789 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:15.036262 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:15.051849 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:09:15.229593 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 14:09:15.265457 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:15.278795 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:15.288012 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:09:15.293443 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:15.293918 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:15.308445 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:15.413492 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:09:15.427935 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:15.488260 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:09:15.524408 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:09:15.544924 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:15.553063 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:15.566706 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:09:15.567227 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:15.567544 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.585650 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:15.591642 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:15.783865 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:15.786666 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:15.787062 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:15.808938 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.809106 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:15.936196 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:16.024518 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:16.035843 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:16.051352 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:16.066669 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:16.066675 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:16.260854 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:16.274417 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:16.293350 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:16.300107 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:16.310963 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:16.310948 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:16.314390 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:16.390187 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:09:16.485576 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:09:16.563218 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:16.576898 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:16.583328 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:16.598889 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:16.602893 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:16.749089 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 14:09:16.784360 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:16.819506 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:16.827170 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:16.847249 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:16.848431 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:16.865311 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:16.895493 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:09:16.903363 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:09:17.025266 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 14:09:17.101083 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:17.102411 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:17.119971 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:17.120887 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:17.121948 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:17.235244 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:17.258306 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 14:09:17.366313 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:17.367305 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:17.373406 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:17.377975 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:09:17.379302 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:17.403889 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:17.474025 25290 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:09:17.474210 25290 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:09:17.474861 25290 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:09:17.485847 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:09:17.490614 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:09:17.499025 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:09:17.499305 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:09:17.499735 25290 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:17.541105 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:17.648499 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:17.656749 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:17.666605 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:17.674244 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:17.686911 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:17.709828 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:17.730700 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:17.918845 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:17.918917 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:17.919303 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:17.919342 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:17.922107 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:17.942083 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:18.034163 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:09:18.081712 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:09:18.095062 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:18.153262 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:18.155284 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:18.156782 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:18.170393 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:18.186420 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:09:18.188779 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:18.398952 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:18.403678 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:18.414850 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:18.432585 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:18.435051 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:18.499557 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:09:18.636773 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:18.641043 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:18.647423 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:18.662019 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:18.673384 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:18.701277 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:18.780004 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:09:18.850351 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:09:18.905583 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 14:09:18.918010 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:18.939150 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:18.950067 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:18.954049 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:18.967482 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:19.078361 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 14:09:19.168370 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.179033 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.183171 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:19.195591 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:19.198930 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:19.243468 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:19.244894 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:19.246456 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:19.407402 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:19.435202 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 14:09:19.444095 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.446007 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:19.456578 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:19.457988 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:19.670198 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.676113 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:19.695142 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.700814 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:19.728549 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:19.755725 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:09:19.787307 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:19.801278 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 14:09:19.810853 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 14:09:19.896564 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:19.909637 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.913977 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:19.929613 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:19.932677 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:19.938663 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:19.970841 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:09:20.072791 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:09:20.073700 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:20.140216 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:20.153802 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:20.166388 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:20.169852 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:20.195050 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:20.264107 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:09:20.324155 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:09:20.383862 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:20.399947 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:20.415386 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:20.421837 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:20.423159 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:20.427886 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:20.460779 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:09:20.531184 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:20.663524 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:20.674079 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:09:20.675124 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:20.678597 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:20.691689 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:20.694182 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:20.704996 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:20.826941 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:20.915807 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:20.926193 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:20.941967 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:20.942839 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:20.943708 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:21.101042 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:21.148538 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:21.153571 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.155045 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.179718 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:21.189587 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:21.219494 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:21.233485 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:09:21.257625 21125 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:09:21.257854 21125 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:09:21.258924 21125 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:09:21.260318 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:09:21.261200 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:21.261317 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:21.261564 21125 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:21.351032 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:21.383772 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:21.399538 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.410650 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.416154 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:21.426812 26217 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:21.427021 26217 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:21.427935 26217 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:09:21.430950 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:21.431005 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:09:21.431233 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:09:21.439306 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:21.439644 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:21.439741 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:21.440332 26217 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:21.623528 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.625701 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.635205 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:21.640231 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:21.648573 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:21.739496 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 14:09:21.826112 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:21.853519 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.855377 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:21.874467 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:21.881330 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:21.883509 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:21.900099 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:22.085934 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:22.091425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.092173 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.105747 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:22.115954 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:22.201584 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:09:22.276713 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 14:09:22.302791 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:22.389143 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.394712 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:22.398262 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 14:09:22.400091 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:22.416999 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:22.422725 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:22.463397 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:22.492431 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:22.622020 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:09:22.629904 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.633318 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:22.635627 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.655081 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:22.658102 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:22.663414 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:22.842880 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:22.870613 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.872567 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:22.889545 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:22.891701 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:22.894591 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:22.987197 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:09:22.996824 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:23.037732 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:23.102773 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.116868 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.118731 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:23.165676 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:23.165958 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:23.294529 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:23.365149 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.370513 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.388309 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:23.412875 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:23.425591 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:23.444483 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:23.623049 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.628334 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.634294 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:23.643292 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:23.658484 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:23.713115 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:09:23.869693 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.870721 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:23.880990 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:23.884667 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:23.899703 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:23.900864 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:23.935617 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:09:23.983568 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 14:09:24.027173 28589 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.16.36.136:36399, user_credentials={real_user=slave}} blocked reactor thread for 43598.6us
W20240501 14:09:24.028301 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fb04ec77da194832aa63d69aba307de3: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:09:24.111682 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:09:24.114169 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240501 14:09:24.172797 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.193837 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:24.195914 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:09:24.200673 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:24.216038 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:24.238462 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:24.249742 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:09:24.425487 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:24.445735 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:24.454010 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.457085 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:24.467171 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.471719 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:24.500600 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:24.586079 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 14:09:24.586212 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:09:24.650741 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:09:24.708220 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.712571 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.723402 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:24.736040 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:24.736600 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:24.928624 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240501 14:09:24.938215 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.946990 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:24.947185 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:24.969434 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:24.975137 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:25.003585 21125 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:25.003805 21125 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:25.004859 21125 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:09:25.005578 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:09:25.005867 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:09:25.007761 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:25.009706 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:25.009848 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:25.012845 21125 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:25.032070 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:09:25.062872 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:25.103760 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:09:25.169723 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:25.171241 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.184243 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:25.190725 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:09:25.198809 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:25.202502 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:25.203524 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:25.250010 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:09:25.322466 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240501 14:09:25.430155 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.431162 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.442447 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:25.459363 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:25.461124 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:25.473603 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:25.567965 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:25.591244 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:25.693390 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.699698 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:25.704478 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.713236 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:25.718940 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:25.824863 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:25.892194 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:25.893707 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:25.915087 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:09:25.918892 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:25.935065 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:25.949667 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:26.015640 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:26.061630 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:26.075942 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:26.094862 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:26.141096 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:26.149142 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:26.160898 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:26.178927 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:26.195201 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:26.397661 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:26.398435 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:26.411082 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 14:09:26.420439 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:26.425379 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:26.451534 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:26.469017 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:26.653059 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:26.653476 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:26.657265 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:26.659837 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:26.672668 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:26.787858 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:26.897140 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:26.897403 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:26.899320 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:26.906967 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:26.913563 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:26.962596 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 14:09:26.994257 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:09:26.996497 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 14:09:27.072265 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:27.076402 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 14:09:27.133028 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.149013 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:27.149648 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.153971 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:27.170845 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:27.360368 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 14:09:27.394876 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.400238 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.407310 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:27.426455 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:27.428037 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:27.573688 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:27.623070 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.635445 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.638734 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:27.644552 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:27.656098 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:27.671425 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:27.675853 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:27.697682 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:27.795562 27191 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:27.795778 27191 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:09:27.796710 27191 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:09:27.797333 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:09:27.797947 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:09:27.798115 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:27.798229 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:09:27.798475 27191 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:27.875353 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:27.885617 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.887310 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:27.916805 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:27.920986 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:27.968545 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:09:28.061292 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:28.131431 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:28.131842 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.133391 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:28.144198 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:28.145817 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.162542 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:28.178933 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:28.359791 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.361069 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.374915 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:28.383926 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:28.384714 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:28.554157 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:28.586900 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.602694 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.617988 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:28.617971 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:28.629267 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:28.651926 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:09:28.679824 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:28.816710 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:28.853286 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.868574 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:28.877955 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:28.882962 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:28.900278 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:29.087363 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:29.089716 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 14:09:29.126516 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:29.127215 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.161394 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.183436 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:29.194394 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:29.196277 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 14:09:29.197429 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:09:29.367436 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 14:09:29.408996 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.415174 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:29.428588 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.432693 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:29.447331 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:29.468113 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:29.518258 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:09:29.553998 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:29.596843 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 14:09:29.620710 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:29.681092 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.685559 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.687254 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:29.703568 27573 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:29.703787 27573 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:09:29.704546 27573 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:09:29.706280 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:29.706509 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:29.715378 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:29.720746 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:29.721186 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:29.721465 27573 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:29.905742 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.916899 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:29.927444 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:29.959175 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:09:29.971693 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 14:09:29.985402 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:30.007176 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:09:30.015604 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:30.024463 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:09:30.036474 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:30.177264 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:30.179590 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:30.206612 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:30.210937 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:30.224714 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:30.327937 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:09:30.386175 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:09:30.398062 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:09:30.426098 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:30.432148 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:30.456765 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:30.469898 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:30.500761 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:30.541966 27573 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:30.542193 27573 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:30.543282 27573 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:09:30.545578 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:30.558585 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:30.563153 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:30.564205 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:09:30.564443 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:09:30.569458 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:30.569700 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:30.570309 27573 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:30.584848 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:30.706020 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:30.710407 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:30.718876 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:30.726979 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:30.742774 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:30.904327 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:09:30.960845 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:09:30.963019 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:30.963973 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:30.988534 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:30.995241 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:31.019681 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:31.118243 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:31.206140 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:09:31.220321 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.223624 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.232753 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:31.248396 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:31.267333 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:31.477001 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:31.484165 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:31.486299 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:31.497938 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.504237 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:31.512118 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:31.611878 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:09:31.648936 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:31.715714 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:31.717331 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.722205 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:31.723740 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.741030 27797 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:09:31.741247 27797 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:09:31.742025 27797 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:09:31.745692 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:09:31.745918 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:09:31.746003 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:09:31.747005 27797 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:31.748821 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:31.946369 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:09:31.953578 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.955562 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:31.979609 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:31.979692 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:31.996548 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:32.000772 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:09:32.051421 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:09:32.051749 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:32.105839 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:09:32.106961 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:32.228729 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:32.232515 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:32.237326 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:32.256132 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:32.262063 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:32.266930 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 14:09:32.350957 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:09:32.371572 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:09:32.416421 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:09:32.488250 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:32.493145 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:32.503301 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:32.507732 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:32.527029 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:32.558499 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:32.614200 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:09:32.645923 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240501 14:09:32.770797 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:32.792935 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:32.807649 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:32.808105 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:32.829676 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:32.953323 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:33.035414 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:33.039655 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:09:33.042066 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:33.059583 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:33.065649 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:33.084452 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:33.107174 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:09:33.113261 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:09:33.257206 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:33.263464 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:33.267374 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:33.276190 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:33.295588 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:33.366482 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:33.537775 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:33.537899 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:33.569909 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:33.570936 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:33.587535 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:33.589888 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:09:33.749095 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:33.778297 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:33.791426 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:33.796801 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:33.811688 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:33.831188 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:33.836143 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:09:33.949721 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:09:33.965763 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:09:34.059371 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.061149 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.062767 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:34.065232 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.071446 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:34.261471 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:34.303850 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.303881 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.307003 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.308233 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:34.328459 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:34.439280 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:09:34.495888 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:09:34.562434 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:09:34.587872 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.593626 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:34.611359 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 48927.1us
W20240501 14:09:34.612846 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 14:09:34.613746 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.616087 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:34.640913 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:34.710706 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:34.723209 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:09:34.724141 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:09:34.806727 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.827811 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:34.833626 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:34.837159 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.839334 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:34.895004 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:09:34.981454 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 14:09:35.001291 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:09:35.055944 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:35.060775 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.062887 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.066375 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:35.088824 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.089728 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:35.162839 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:35.300478 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.302605 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.324187 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:35.336118 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:09:35.339494 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.355255 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:35.393121 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:09:35.528404 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240501 14:09:35.558516 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:35.563279 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:35.575412 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.589565 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.631799 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:35.686297 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:35.696005 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:35.807562 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:09:35.833914 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.834883 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.864207 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:35.875978 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.880689 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:35.902820 27573 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:35.902995 27573 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:09:35.907629 27573 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
W20240501 14:09:35.911407 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:09:35.911327 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:09:35.913152 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:09:35.914407 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:35.914553 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:09:35.918577 27573 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:36.100723 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.105711 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.111441 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:36.122092 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:36.124105 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:36.195060 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:09:36.210371 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:09:36.216143 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:09:36.217067 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:36.329025 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.358848 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:36.365007 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.370059 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:36.388270 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:36.422335 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:09:36.506534 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:09:36.581650 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.590909 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.607832 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:36.618849 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.627527 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:36.735179 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:36.816205 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.831429 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:36.839551 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:36.846352 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.859601 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:36.876120 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:37.053707 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 14:09:37.069552 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.076053 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.077739 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:37.100600 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.113615 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:37.211022 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:09:37.295197 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:37.308563 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.314363 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:37.328358 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:37.348021 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:37.351135 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 14:09:37.356174 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.364907 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:37.378885 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:09:37.446442 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:09:37.526345 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 14:09:37.567363 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.568325 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:37.571043 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.591956 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.604480 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:37.706018 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:09:37.799686 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:37.813164 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:37.817571 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:37.826593 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:37.840440 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 14:09:37.883522 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:37.912964 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:38.075232 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:38.090744 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.091037 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.096575 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.111591 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.133198 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.346477 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.369665 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.389345 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.390033 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.391266 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:38.451431 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:38.511920 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:09:38.610188 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:38.612270 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.614665 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.630766 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.633373 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.634492 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.699644 21125 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:38.699865 21125 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:38.700760 21125 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:09:38.702386 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:09:38.702680 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:09:38.705919 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:09:38.706241 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:38.706367 30862 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:38.706658 21125 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:38.834786 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:38.846385 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:38.869254 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:38.871452 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:38.875577 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:38.887773 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:39.041990 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:09:39.079912 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.089011 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.096765 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:39.112196 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:39.118235 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:39.135448 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:09:39.212510 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:39.325412 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.327939 29385 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:39.328204 29385 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:09:39.328918 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.329231 29385 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:09:39.331571 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:09:39.332681 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:09:39.338058 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:39.341883 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:39.342145 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:09:39.343098 29385 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:39.368266 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:39.376119 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:39.380168 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:39.393625 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:09:39.468948 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:09:39.472754 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:09:39.584117 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.590701 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.625710 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:39.627941 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:39.644153 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:39.760156 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:09:39.813320 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:09:39.814775 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:09:39.892964 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.895148 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:39.908898 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:39.921095 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:39.931226 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:39.936439 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:39.989295 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:09:40.012113 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 14:09:40.020840 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:40.158926 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.166218 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:40.169438 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:40.179509 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:40.186332 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:40.213595 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:40.289547 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:09:40.331732 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:40.384138 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 14:09:40.392689 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:09:40.404502 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:09:40.414544 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.417850 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:40.439352 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.452390 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:40.459107 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:40.471169 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 14:09:40.663745 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.669344 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.679216 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:40.688671 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:40.692517 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:40.825430 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:40.866292 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:09:40.904706 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:40.946480 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.951572 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:40.979069 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:40.996243 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:41.058856 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:09:41.144327 29773 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:09:41.144551 29773 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:09:41.145434 29773 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:09:41.148149 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:41.148388 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:09:41.148511 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:41.148811 29773 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:41.221863 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:41.223495 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.223737 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:41.236131 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:09:41.237455 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:41.238035 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:41.307627 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:41.489979 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.490988 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.494645 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:41.512631 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:41.526165 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:41.569617 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:09:41.610250 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:41.701225 29385 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:41.701439 29385 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:41.702414 29385 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:09:41.704907 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:41.712267 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:09:41.712574 30927 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:09:41.712816 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:41.712973 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:41.713297 29385 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:41.739861 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.743713 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.744199 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:41.768954 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:41.783797 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:41.789886 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:09:41.803123 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:09:41.879809 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:41.934485 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:09:41.954432 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:41.985003 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:41.987289 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.006593 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:42.009603 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:42.053201 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:42.123832 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:09:42.231227 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.240264 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.241050 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:42.247906 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:42.253594 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:42.316937 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:09:42.368537 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:09:42.368959 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:42.455372 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:09:42.465035 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240501 14:09:42.475582 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.483868 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.492676 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:42.496615 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:42.498980 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:42.593056 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 14:09:42.702436 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:42.721210 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.722292 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.725986 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:42.735244 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:42.752622 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:42.841572 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:09:42.854290 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:42.981958 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:42.982393 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.985160 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:42.995165 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:42.999706 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.221171 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:43.225212 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.231508 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.250865 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.254954 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:43.306521 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:43.307286 29773 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:09:43.307472 29773 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:09:43.308225 29773 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:09:43.324509 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:09:43.324762 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:43.324879 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:43.325224 29773 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:43.326670 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:43.357610 30240 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:43.357885 30240 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:09:43.358930 30240 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:09:43.362073 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:43.372329 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:43.372470 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:43.372753 30240 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:43.523218 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.529690 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:43.533416 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.534076 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:43.552953 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:43.760520 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:43.767027 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.772109 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:43.809085 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:43.816177 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:43.834082 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:09:43.840324 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:43.868366 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:43.993438 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.017618 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.018756 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:44.030920 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:44.057195 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:44.073969 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:09:44.234282 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.246385 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.247575 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:44.267226 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:44.279485 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:44.339221 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:09:44.388370 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:44.436223 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:09:44.478121 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:09:44.487665 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.499961 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:44.502729 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:09:44.510839 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:44.530963 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:44.535187 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:44.578011 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:44.766031 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.769868 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:44.780586 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:44.791162 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:44.801501 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:44.939472 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:44.975699 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:44.976248 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:09:44.982161 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:09:44.989403 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240501 14:09:45.025597 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.035070 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.054206 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:45.059502 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:45.069615 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:45.155947 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:09:45.169121 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 14:09:45.175827 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 14:09:45.195245 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:09:45.250221 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240501 14:09:45.251478 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.258738 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.267845 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:45.285043 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.288024 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:45.412796 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:09:45.424571 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:45.538661 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.548866 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.559693 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:45.573760 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.575728 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:45.627820 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:09:45.778776 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.791123 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:45.808918 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:45.827962 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:09:45.830859 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:45.837127 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:46.000133 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:46.049088 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.055788 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:46.060506 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.074561 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:46.079658 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:46.110570 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 14:09:46.294701 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:46.307085 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:46.313930 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:09:46.318961 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.343719 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:46.363358 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:46.434418 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:09:46.482641 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:46.582369 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.585019 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:46.615509 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.618387 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:46.619379 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:09:46.628659 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:46.816169 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:09:46.826558 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:09:46.833570 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.847019 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:46.855162 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:46.860419 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:46.879603 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:46.893421 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:47.022830 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:09:47.104224 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.108479 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.126940 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:47.130731 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:47.132023 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:09:47.159201 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:47.253551 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:09:47.258345 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:09:47.318840 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:47.338244 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:09:47.397779 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.403295 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.413026 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.419917 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:47.425482 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:47.492298 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 14:09:47.583213 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 14:09:47.641665 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.652261 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:47.655118 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.670315 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:47.674863 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:47.727919 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:09:47.745059 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:47.752362 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:09:47.805603 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 14:09:47.909898 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.916456 29773 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:09:47.916666 29773 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:09:47.916834 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:47.917279 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:47.917377 29773 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:09:47.930673 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:47.935232 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:09:47.935459 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:47.935561 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:47.935815 29773 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:47.946987 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:48.149168 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:48.153360 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:48.154170 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:48.161116 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:48.168984 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:48.243880 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:48.329939 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:09:48.380929 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:48.383702 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:48.401019 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:48.404785 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:48.424106 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:48.617033 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:48.627027 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:48.631582 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:48.645339 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 14:09:48.650430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:48.652671 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:48.748353 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:48.873834 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:48.887148 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:48.889550 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:48.894286 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:48.899767 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:48.995636 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:09:49.107466 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:09:49.126466 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:49.135737 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:49.136173 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:49.137298 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:49.137905 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:49.138931 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:49.303385 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:09:49.389519 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:49.398034 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:49.399580 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:49.407390 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:49.413342 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:49.429591 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:09:49.452216 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:09:49.663779 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:49.674286 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:49.677313 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:49.691639 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:49.699282 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:49.711763 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:49.718538 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:09:49.718986 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:49.736434 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:49.923957 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:09:49.957991 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:49.965642 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:49.970163 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:49.983152 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:49.984220 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:50.026348 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:09:50.036214 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:50.051311 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:09:50.096545 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:50.215735 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:50.219439 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:50.229832 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:50.258401 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:50.265894 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:50.276314 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 14:09:50.286192 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 14:09:50.372520 30240 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:50.372751 30240 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:50.373641 30240 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:09:50.374670 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:09:50.374928 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:09:50.375154 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:09:50.375486 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:50.375622 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:50.375906 30240 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:50.467859 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:09:50.505746 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:50.512575 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:50.514467 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:50.518572 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:50.520606 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:50.576438 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:09:50.603504 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:09:50.625610 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:50.665745 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:09:50.744927 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:50.745774 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:50.753710 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:50.770130 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:50.771895 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:50.775259 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:09:50.798898 31633 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:09:50.799126 31633 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:09:50.800000 31633 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:09:50.806813 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:09:50.807370 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:50.807507 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:09:50.807777 31633 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:50.989781 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:50.990239 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:50.999171 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:09:50.999984 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:51.031033 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:51.032339 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:51.135131 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:51.238889 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:51.240689 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:51.256485 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:51.258965 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:51.262142 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:51.356631 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:51.431800 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:09:51.464687 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:09:51.497224 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:51.503252 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:51.503253 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:51.511562 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:51.521909 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:51.667869 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:51.729868 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:51.779853 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:51.784637 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:51.788311 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:51.818993 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:51.823545 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:51.933247 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:51.973567 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:09:52.058634 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.071283 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:52.074563 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.076217 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:52.096973 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:52.127044 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:52.175105 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:52.287590 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:52.299724 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:52.303658 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.320789 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.342113 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:52.358907 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:52.383802 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:52.415700 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:09:52.484475 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:09:52.553005 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.556813 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.577872 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:52.577900 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:52.578938 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:52.741580 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:09:52.748905 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 14:09:52.789731 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:52.822950 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:52.823698 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.826017 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:52.827814 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:52.847755 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:52.912933 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240501 14:09:53.073079 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.077026 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.078258 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:53.083801 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:53.098942 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:53.111337 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:53.305541 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:09:53.317205 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.318949 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:53.321182 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.340420 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:53.344689 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:53.373539 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:53.563311 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.581169 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.583951 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:53.585438 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:53.587528 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:53.603049 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:09:53.850090 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.865372 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:53.884626 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:53.885577 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:53.892765 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:53.904546 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:53.939810 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:54.039201 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:54.137385 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:09:54.164368 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.175241 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:54.179155 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.196239 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:54.208290 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:54.240617 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:54.364770 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:54.402251 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.406534 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.411984 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:54.441963 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:54.482312 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:54.520437 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:54.549294 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:54.578047 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:54.661800 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.665457 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.673749 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:54.702358 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:54.719151 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:54.800822 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:54.868443 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:54.895052 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:54.911880 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.913420 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:54.922438 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:54.927341 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:54.940181 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:55.037161 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:09:55.104796 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:09:55.104966 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 14:09:55.135423 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:09:55.213939 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:55.222942 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:55.224117 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:55.248445 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:55.253624 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:55.391222 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:55.422050 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:09:55.489012 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:55.489526 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:55.499455 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:55.510002 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:55.518980 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:55.677421 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:09:55.726171 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:55.739133 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:55.746712 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:55.749338 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:55.753883 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:55.758111 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:55.760289 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:55.827733 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 14:09:55.831041 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:09:55.881301 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:56.040740 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.056449 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:56.057956 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:56.067267 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.086287 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:56.305855 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.318459 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:56.324234 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.339510 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:56.352953 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:09:56.354136 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:56.373208 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:56.456501 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:09:56.472339 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:09:56.473245 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:56.603464 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.613471 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.622316 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:56.622925 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:56.626544 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:09:56.636528 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:56.657742 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:56.837571 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.841444 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:56.861366 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:56.863006 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:56.867946 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:56.975759 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:56.993921 32306 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:56.994131 32306 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:09:56.995136 32306 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:09:56.996538 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:09:56.997450 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:09:56.997833 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:56.997947 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:09:56.998453 32306 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:57.021867 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:09:57.041136 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:57.073127 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:57.087723 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:57.088840 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:57.093279 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:57.094484 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:57.119177 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:09:57.274363 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:57.321367 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:57.342576 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:57.348029 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:57.353767 32306 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:57.354003 32306 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:09:57.354993 32306 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:09:57.358202 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:09:57.358531 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:09:57.359258 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:09:57.379820 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:57.380542 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:57.382102 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:09:57.382395 32306 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:57.394699 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:57.515385 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:57.551275 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 14:09:57.564646 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:09:57.579685 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 14:09:57.646374 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:57.649526 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:57.652618 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:57.659862 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:57.696652 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:57.871024 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 14:09:57.874339 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:09:57.883424 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:57.892099 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:57.913813 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:57.924132 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:57.926576 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:57.975817 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:09:57.976137 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:58.173986 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.192349 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.202112 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:58.212309 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:58.220535 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:58.344733 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:58.349289 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:09:58.428527 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.436940 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.441304 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:58.454712 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:58.460584 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:58.564163 32306 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:09:58.564369 32306 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:09:58.565060 32306 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:09:58.566794 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:09:58.568853 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:09:58.569141 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:09:58.569239 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:09:58.569773 32306 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:58.689976 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.695875 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.700137 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:58.713287 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:58.738236 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:09:58.738422 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:09:58.779616 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:58.903838 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:58.937345 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.942355 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:58.951920 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:58.955705 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:58.959179 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:09:58.992482 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:59.039165 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:09:59.186844 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.191304 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.194793 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:59.212986 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:59.213995 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:59.314587 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:09:59.315876 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:59.366137 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:09:59.455221 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.464635 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.477424 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:59.480615 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:59.493710 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:09:59.630803 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:09:59.670451 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:09:59.678519 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.692078 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:59.711313 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:59.711949 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.730338 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:09:59.776225 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:09:59.812166 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:09:59.816993 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:09:59.908769 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 14:09:59.937124 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.938153 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:59.956220 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:59.966935 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:59.992678 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:00.210667 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:00.215837 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:00.223243 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:10:00.228657 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:00.250313 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:00.251488 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:00.319617 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 14:10:00.325400 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:10:00.386999 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:00.395952 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:10:00.490589 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:00.498713 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:00.499714 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:00.503803 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:00.505813 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:00.737448 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:00.739352 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240501 14:10:00.746241 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:00.747481 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:00.759904 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:00.778110 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:00.797293 32306 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:00.797535 32306 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:10:00.798319 32306 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:10:00.800791 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:00.815088 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:00.815366 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:00.815714 32306 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:00.817044 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:10:00.840332 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:10:00.846982 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:00.986289 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:01.007006 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.021942 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.035228 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:01.051353 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:01.071725 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:01.101759 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:10:01.266395 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.266857 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:01.268889 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.274536 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:01.299453 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:01.367064 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:10:01.469513 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:10:01.494278 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 14:10:01.542876 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:01.545164 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:10:01.545693 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.573619 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:01.592446 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:01.602413 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:01.615427 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:01.634424 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:10:01.693799 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:10:01.716044 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:10:01.807737 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.820591 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:01.838970 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:01.853173 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:01.925642 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:02.029706 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:10:02.097059 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:02.104200 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:02.119925 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:02.122932 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:02.126422 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:02.144272 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:02.164233 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:10:02.333876 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:02.340085 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:02.343326 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240501 14:10:02.442106 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:02.443768 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:02.476787 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:02.480867 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:10:02.557287 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:02.576781 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 14:10:02.613802 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:10:02.658354 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:02.661111 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:02.664036 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:02.687086 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:02.690230 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:02.693500 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:10:02.933116 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:02.940541 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240501 14:10:02.948761 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:02.949698 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:02.952499 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:02.953658 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:02.979122 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:03.213213 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:10:03.219166 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:03.228404 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:03.260138 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:03.261557 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:03.262694 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:03.494019 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:10:03.516496 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:03.527659 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:03.541779 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:03.542802 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:03.543072 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:03.604132 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:03.690659 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:10:03.788492 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:03.792606 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:03.800729 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:03.801360 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:03.816114 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:03.975793 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:10:04.060933 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.062018 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.074797 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:04.075670 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:04.103027 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:04.104071 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:10:04.106495 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:04.139695 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:10:04.245842 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:10:04.295043 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.302078 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.347198 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:04.354871 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:04.363307 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:04.373183 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:04.436211 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:10:04.611308 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.616247 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:04.622973 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.634076 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:04.639310 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:04.686995 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 50866.3us
W20240501 14:10:04.739961 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:04.912853 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:04.927279 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:04.929826 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:04.945372 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:10:04.959435 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:04.964067 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 14:10:04.970337 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:05.066931 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 14:10:05.100574 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:10:05.130164 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:05.176549 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:05.189227 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:05.198431 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:05.203289 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:05.218617 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:05.220808 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:05.246692 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:10:05.280649 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 14:10:05.432500 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:05.440418 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:05.456290 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 14:10:05.467394 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:05.467411 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:05.470973 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:05.532804 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:05.686834 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:05.698484 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:05.739982 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:05.747464 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:05.754931 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:05.851466 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:05.938856 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:10:05.996641 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:06.009864 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:06.018863 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:06.019438 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:06.048641 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:06.061111 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:06.112324 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240501 14:10:06.178344 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:10:06.240432 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:06.247113 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:06.247573 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:06.259876 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:06.264484 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:06.311321 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:10:06.473435 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:10:06.473505 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:10:06.502918 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:06.506886 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:06.517551 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:10:06.521401 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:06.530928 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:06.555982 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:06.582445 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:06.661521 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:10:06.726576 1509 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:06.727275 1509 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:10:06.728850 1509 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:10:06.732414 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:06.733539 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:06.733676 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:06.733983 1509 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:06.790462 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:10:06.803503 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:06.817310 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:06.827124 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:06.833215 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:06.865638 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:06.869268 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:06.884475 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:10:06.892532 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:06.937577 1509 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:06.937799 1509 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:10:06.938925 1509 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:10:06.940793 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:10:06.941040 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:10:06.943652 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:06.944034 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:06.944167 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:06.945243 1509 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:07.042754 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:07.085402 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.102670 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:07.103683 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.110678 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:07.133430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:07.309674 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.311511 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.320731 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:07.351007 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:07.352478 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:07.365299 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:10:07.454089 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:10:07.473407 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 14:10:07.481137 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240501 14:10:07.482506 446 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:10:07.482728 446 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:10:07.483620 446 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:10:07.492456 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:10:07.492681 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:07.492781 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:07.493038 1929 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:07.567427 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:07.578117 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:07.579968 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:07.580423 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240501 14:10:07.584036 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.589934 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:07.614694 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:07.643936 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:07.701289 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:07.793146 1463 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:10:07.793359 1463 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:10:07.794358 1463 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:10:07.796597 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:07.802973 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:07.805857 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:10:07.806067 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:10:07.806380 1463 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:07.844681 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:07.847560 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:07.863072 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:07.865355 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:08.098671 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.112434 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.119560 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:08.129112 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:08.130023 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:08.219862 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:08.371613 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.385505 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.386900 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:08.388424 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:08.410774 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:08.466706 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:10:08.631222 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:10:08.638581 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.649223 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.667791 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:08.674959 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:08.681464 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:08.689960 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:08.877908 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:10:08.899286 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.903964 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:08.925539 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:08.926539 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:08.941233 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:09.077490 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:10:09.099566 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:09.140949 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.156407 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.163249 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:09.164027 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:09.189002 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:10:09.189018 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:09.236570 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:10:09.286579 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:09.383226 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.383363 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.395102 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:09.400417 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:09.426003 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:09.454490 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:10:09.526597 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:10:09.628854 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.639456 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.647753 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:09.653029 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:09.671922 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:09.696482 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:09.729539 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 14:10:09.877333 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.883482 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:09.894150 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:09.899765 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:09.904278 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:09.954298 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:10:09.985935 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 14:10:10.010514 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:10:10.032696 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:10:10.070001 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 14:10:10.164994 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:10.168553 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.169672 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.187654 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:10.212335 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:10.300820 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:10.379355 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:10:10.408305 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:10:10.413643 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.414489 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.432798 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:10.442070 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:10.486758 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:10.670974 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:10.671034 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.682113 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.704741 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:10.718212 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:10.849043 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:10.899842 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:10.900869 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:10.905534 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 14:10:10.908272 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:10.908727 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:10.911460 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:11.016532 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:11.154558 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.159030 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.160986 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:11.181246 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:11.183713 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:11.248683 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:10:11.271534 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:11.309743 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:11.325098 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:10:11.396660 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.411696 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:11.417392 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.421553 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:11.438549 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 34 similar messages]
I20240501 14:10:11.443346 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:11.506806 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:10:11.569245 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:10:11.601907 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:10:11.654186 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.658818 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.663640 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:11.668241 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:11.675005 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:11.696023 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:11.707127 2134 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:10:11.707365 2134 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:10:11.708137 2134 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:10:11.710754 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:10:11.711122 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:11.711258 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:10:11.712119 2134 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:11.733449 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:10:11.887223 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.898444 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:11.917160 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:11.918946 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:11.927919 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:11.939754 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 14:10:12.098850 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:12.123160 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:10:12.172979 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.181159 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.185647 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:12.187230 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:12.188222 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:12.255057 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:10:12.341243 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:12.345670 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:10:12.400504 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.423661 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.429126 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:12.429631 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:12.440119 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:10:12.469213 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:12.477237 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:12.492777 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:10:12.607353 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 14:10:12.644981 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 14:10:12.661998 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.668164 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.678726 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:12.691766 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:12.693388 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:12.915294 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:12.919023 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:10:12.919989 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:12.940732 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:12.956260 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:12.978444 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:13.071820 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:13.171808 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.187399 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.195392 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:13.208899 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:13.213951 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:13.448398 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.456494 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.462965 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:13.463507 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:13.481648 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:13.485445 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:13.583763 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:10:13.655337 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:10:13.681416 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.703423 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.706535 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:13.721104 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:13.723958 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:13.774865 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:13.816427 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:10:13.822021 2832 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:13.822240 2832 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:10:13.823575 2832 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:10:13.824857 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:10:13.825135 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:10:13.825418 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:13.827062 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:13.827203 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:13.827517 2832 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:13.954411 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.962486 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:13.963743 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:13.969388 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:14.015537 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:14.064811 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:14.077032 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:10:14.149327 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:10:14.202788 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.205725 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:14.211638 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:14.216109 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.221208 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:14.291653 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:10:14.339450 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:10:14.467962 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.470595 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.471101 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:14.486699 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:14.503563 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:14.538152 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:14.548523 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:14.705096 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.717811 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.720911 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:14.729147 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:14.739220 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:14.769395 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:14.947558 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:10:14.950747 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:14.965232 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:14.983536 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:14.986899 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:14.991428 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.007499 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:15.016261 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:15.026744 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:15.160245 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 14:10:15.182859 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240501 14:10:15.234282 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.243261 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:15.246495 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.247831 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:15.261474 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:15.442005 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:15.452904 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:10:15.506618 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.509406 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:15.514851 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.520711 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:15.535223 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:15.752506 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:10:15.764418 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.768679 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:15.773468 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:15.781831 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:15.801291 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:15.868368 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:15.918572 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:15.980201 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:10:16.017484 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.027128 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.038604 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:16.059113 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:16.084277 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:16.193109 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:16.264267 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.268018 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.277519 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:16.296536 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:16.308461 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:16.425410 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:10:16.470901 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:16.494685 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:10:16.505604 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:10:16.517428 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.520790 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:16.527392 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:10:16.540899 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:16.546242 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:16.547374 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:16.661738 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:10:16.757979 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:10:16.780396 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:16.793051 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:16.799270 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:16.800561 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:16.819200 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:16.831310 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:10:16.861292 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:10:16.863822 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:16.868438 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:10:17.019680 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.029135 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.050333 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:17.054081 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.076179 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:17.082404 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 14:10:17.237526 3059 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:17.237736 3059 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:10:17.238765 3059 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:10:17.239835 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:10:17.240099 30927 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:10:17.240252 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:17.240664 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:17.240758 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:17.241017 3059 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:17.287174 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:17.287886 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:17.307873 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:10:17.311098 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 14:10:17.314690 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:17.318972 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:17.325568 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.344952 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:17.370841 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240501 14:10:17.388382 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:17.434821 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:17.600728 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.610792 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:17.612562 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.651516 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.654857 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:17.687018 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240501 14:10:17.838235 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.845767 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:17.852193 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:17.861866 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:17.868198 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:17.882958 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:18.011858 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:18.056574 28593 maintenance_manager.cc:419] P 1bdce3de4d694d6c98f909a5b3c450ce: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033343
I20240501 14:10:18.072505 28592 maintenance_manager.cc:643] P 1bdce3de4d694d6c98f909a5b3c450ce: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.007s sys 0.004s Metrics: {"bytes_written":2209,"delete_count":0,"lbm_write_time_us":286,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":4818,"threads_started":1,"update_count":35,"wal-append.queue_time_us":4930}
I20240501 14:10:18.073328 28593 maintenance_manager.cc:419] P 1bdce3de4d694d6c98f909a5b3c450ce: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.151936
I20240501 14:10:18.083794 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.084228 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.087548 3059 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:18.087723 3059 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:10:18.088587 3059 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:10:18.096935 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:10:18.097609 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:10:18.103965 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:18.104135 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:10:18.104462 3059 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:18.105027 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.114954 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:18.169811 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:18.219888 28592 maintenance_manager.cc:643] P 1bdce3de4d694d6c98f909a5b3c450ce: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.146s user 0.026s sys 0.008s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14883,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":8657,"dirs.run_cpu_time_us":352,"dirs.run_wall_time_us":2331,"lbm_read_time_us":568,"lbm_reads_lt_1ms":17,"lbm_write_time_us":1102,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":14617,"threads_started":2,"update_count":35}
I20240501 14:10:18.388561 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:18.389261 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.400516 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.401881 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.417855 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:18.453248 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:18.595095 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:10:18.633728 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.636127 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:18.637744 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.645313 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.656481 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:18.663456 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:10:18.841094 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.852058 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:18.900709 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:18.900825 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:18.910704 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:18.999948 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:19.089766 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:19.158666 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:19.158893 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:10:19.164574 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:19.164570 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:19.168289 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:19.175303 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:19.221052 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:19.234572 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:10:19.399334 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:19.405651 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:19.412698 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:19.417263 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:10:19.423501 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:19.435734 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:19.468643 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:10:19.599998 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 14:10:19.612617 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:19.662762 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:19.669188 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:19.683589 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:19.696905 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:19.714928 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:19.728222 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:19.748224 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:10:19.771183 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:10:19.868893 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 14:10:19.914403 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:19.919661 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:19.925347 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:19.944257 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:19.947752 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:19.980104 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:10:20.170946 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:20.174662 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:20.182406 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:20.195955 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 14:10:20.199785 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:20.203354 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:20.225915 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:20.407219 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:20.409039 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:20.411296 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:10:20.428292 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:20.439417 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:20.449685 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:20.654047 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:20.656363 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:20.668926 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:20.674746 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:20.702831 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:20.715821 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:20.804598 3955 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:20.804790 3955 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:10:20.805511 3955 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:10:20.807322 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:20.807500 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:20.807652 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:20.807919 3955 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:20.913389 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:20.921797 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:20.923925 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:20.942775 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:20.946352 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:21.011163 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:21.012370 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:21.061473 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 14:10:21.170203 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:21.188345 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:21.192185 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:21.197403 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:21.214879 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:21.282861 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:21.318341 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:10:21.406376 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:10:21.500136 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:21.509936 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:21.510586 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:21.510783 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:21.528292 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:21.688347 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:21.701661 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:21.702740 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:21.703485 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20240501 14:10:21.739161 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:21.742401 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:21.749310 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:21.757470 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:21.777321 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:21.783160 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:10:21.802778 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:21.991255 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:21.998391 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:10:22.011322 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:22.012468 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240501 14:10:22.017160 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.023097 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.034446 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:22.036612 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:22.062150 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:22.119000 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:10:22.168541 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:22.183374 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:10:22.238584 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:22.261885 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.275602 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:22.276567 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:22.288203 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.290680 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:22.420117 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240501 14:10:22.528877 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:22.530586 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.536743 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.537029 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:22.556998 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:22.568604 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 14:10:22.728150 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:10:22.799862 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:22.800026 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.812402 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:22.826853 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:22.828133 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:22.849066 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:23.062793 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:23.063558 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:10:23.067970 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.074553 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.090924 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:23.097298 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:23.184985 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:23.321075 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.324803 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.340349 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:23.353302 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:23.354307 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:23.365788 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:10:23.592332 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.599303 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:23.607169 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.636430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:23.637473 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:23.640309 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:23.696774 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:10:23.830034 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.842324 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:23.870683 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:23.888531 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:23.911670 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:23.929872 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:24.053638 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:10:24.069103 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 14:10:24.083256 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fb04ec77da194832aa63d69aba307de3: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:10:24.106151 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.113909 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.120354 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:24.140041 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:24.143851 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:24.148496 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:10:24.167140 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:24.183874 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:10:24.323441 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:10:24.341202 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240501 14:10:24.369159 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.374687 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.424621 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:24.436178 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:24.443245 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:10:24.472748 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:24.484280 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:24.558816 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:10:24.699530 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.702013 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.702911 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:24.715586 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:10:24.716467 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:24.728431 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:24.773546 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:24.930467 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.938966 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:24.941195 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:24.952762 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:24.954707 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:24.958868 2832 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:10:24.959066 2832 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:10:24.959969 2832 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:10:24.993402 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:10:24.993645 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:24.993750 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:24.994016 2832 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:24.995649 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 14:10:25.163796 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240501 14:10:25.178980 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.186282 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.199357 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:25.203752 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:25.221161 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:25.277246 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:10:25.304366 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:25.383697 2832 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:25.383980 2832 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:10:25.385457 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:10:25.385766 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
I20240501 14:10:25.386296 2832 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:10:25.388625 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:25.389343 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:25.389477 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:25.389770 2832 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:25.403609 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:10:25.433427 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.445206 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.459174 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:25.475505 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:25.477823 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:25.692319 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.697716 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.698004 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:25.700865 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:10:25.712426 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:25.715828 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:25.889566 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:25.918999 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:10:25.928697 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.930001 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:25.946025 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:25.955260 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:25.998328 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:26.128590 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:10:26.183602 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:26.192692 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:26.204890 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:26.218044 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:10:26.219205 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:26.229426 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:26.306427 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:10:26.440227 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:26.441282 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:26.441710 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:26.475561 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:26.476795 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:26.479017 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:26.616467 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:10:26.668597 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:10:26.703455 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:10:26.707175 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:26.708956 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:26.719177 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:26.724279 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:26.732755 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:26.750900 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:26.763346 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:10:26.817927 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:10:26.820677 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:10:26.857259 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:26.902262 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:26.960455 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:26.975167 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:26.985914 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:26.991492 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:27.009756 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:10:27.010895 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:27.035322 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:10:27.106989 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:27.168625 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:27.210880 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:27.223948 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:27.231082 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:27.240449 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:27.260107 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:27.349110 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 14:10:27.363493 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:27.483228 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:27.483505 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:27.486876 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:27.491273 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:27.494207 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:27.573635 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:10:27.590055 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 14:10:27.721663 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:27.736138 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:27.737551 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:27.739163 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:27.764081 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:27.770481 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:27.853960 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:27.983312 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:27.998382 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:27.999763 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:28.019759 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:28.044831 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:28.138209 4936 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:10:28.138437 4936 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:10:28.139382 4936 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:10:28.141213 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:10:28.141932 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:10:28.142066 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:10:28.142309 4936 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:28.188944 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:10:28.264487 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:28.271718 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:28.277406 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:28.281286 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:28.307528 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:28.360538 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:28.536720 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:28.540179 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:28.540526 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:28.556411 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:28.563181 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:28.598464 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:10:28.729642 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:10:28.787637 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:28.797441 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:28.797885 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:28.798574 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:28.804605 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:28.820593 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:28.929703 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240501 14:10:29.051678 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:29.061789 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:29.066747 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:29.087564 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:29.089361 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:29.097412 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:29.166337 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:10:29.191449 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:10:29.296257 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:29.306828 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:29.311348 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:29.329535 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:29.330207 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:29.333278 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:10:29.344719 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:29.355926 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:29.461310 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:10:29.521142 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:10:29.574122 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:29.581543 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:29.586970 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:29.599915 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:29.619086 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:29.619220 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:29.678988 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:29.816525 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:29.839315 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:29.840515 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:29.851132 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:29.857072 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:29.861021 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:29.943732 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 14:10:30.064376 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240501 14:10:30.087581 5139 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:30.087801 5139 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:10:30.088639 5139 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:10:30.091746 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:10:30.091356 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:30.093036 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:10:30.093178 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:10:30.093992 5139 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:30.123087 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:30.123821 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:10:30.131776 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:30.134039 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:30.159590 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:30.161029 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:30.317852 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:30.380404 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:30.381172 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:30.388286 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:30.400851 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:30.418390 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:30.635046 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:30.652468 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:30.656112 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:30.679104 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:30.700253 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:30.781927 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:30.782275 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:30.886091 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:30.887347 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:30.903577 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:30.911473 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:30.923694 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:31.072039 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240501 14:10:31.155015 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:31.155033 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:10:31.176380 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:31.181502 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:31.193359 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:10:31.195325 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:31.228473 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:10:31.239660 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:10:31.243850 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:31.311134 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:31.435336 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:31.437809 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:10:31.451364 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:31.456037 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:31.457170 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:31.464895 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:31.624303 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:10:31.626307 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:10:31.674444 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:31.676103 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:31.682313 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:31.682917 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:31.696842 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:31.783099 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:31.811293 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:10:31.858741 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:10:31.887463 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 14:10:31.926565 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:31.937536 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:31.949911 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:31.951077 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:10:31.957290 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:31.969051 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:32.008728 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:10:32.064561 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:10:32.120560 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:32.158320 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:10:32.177392 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 763b2e84afa64924ad5eeeeb3265a1d8: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:32.177533 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:32.186887 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:32.195842 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.196904 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:32.208108 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:32.262651 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:10:32.334492 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:10:32.351286 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:10:32.356294 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:32.466413 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.468135 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:32.498492 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:32.498561 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:32.517261 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:32.555181 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240501 14:10:32.734763 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:32.740468 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:32.742870 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:32.743499 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240501 14:10:32.743849 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:32.745431 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:32.854203 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:32.985404 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.000499 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.005651 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.018947 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:33.037150 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.224992 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.230223 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.239514 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.253949 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:33.256454 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:33.397217 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:33.487967 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:10:33.505023 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240501 14:10:33.559254 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.561987 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.597779 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:33.600204 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.606702 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:33.619433 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:10:33.688189 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:10:33.837388 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:10:33.862600 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.873867 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:33.875830 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:33.892796 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:33.907894 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:33.950300 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:33.975754 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:10:34.122799 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.138994 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.140106 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.159631 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.174235 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:34.229394 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:10:34.266314 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:10:34.273021 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:10:34.336761 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:10:34.375054 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.379165 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.392544 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:34.396230 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.404174 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:34.408743 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:10:34.511569 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:10:34.532241 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:34.604743 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.618077 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.627681 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.654074 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:34.683058 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:34.794466 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:10:34.814188 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240501 14:10:34.875449 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.876951 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:34.882719 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:34.895607 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:34.899442 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:34.966198 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:34.973452 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:10:35.098611 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.100976 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.103881 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.131475 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.182236 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:35.316709 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 14:10:35.343993 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:35.381173 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.381515 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.386793 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.394215 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:35.409025 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.555564 5139 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:35.555783 5139 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:10:35.556623 5139 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:10:35.557188 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:10:35.557821 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:10:35.558367 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:35.558532 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:10:35.558877 5139 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:35.635362 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.639420 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.639613 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.670130 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:35.672036 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.816555 5139 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:35.816817 5139 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:10:35.817873 5139 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:10:35.819833 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:35.820109 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:35.820262 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:35.820590 5139 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:35.830102 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:10:35.868515 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:35.903539 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:10:35.913194 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.916188 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:35.923913 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:35.946677 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:35.947230 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:36.014374 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 14:10:36.018733 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:10:36.180718 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:36.187558 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:10:36.193764 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.199148 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:36.208478 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.210003 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.221060 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:36.318418 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:36.421548 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:36.451058 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:36.451743 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.464073 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:36.464527 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.490378 5548 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:10:36.490574 5548 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:10:36.491390 5548 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:10:36.499372 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:36.513327 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:10:36.513536 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:36.513603 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:36.513832 5548 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:36.623423 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:10:36.691663 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.699507 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:36.703284 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:10:36.706032 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.706811 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:36.731876 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:36.782152 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:10:36.807931 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:10:36.814836 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:36.936174 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:10:36.966421 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.967443 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:36.969363 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:36.985579 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 14:10:36.990860 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:37.022749 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.197805 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:37.207181 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:37.217202 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:37.233106 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.244679 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:37.290187 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:10:37.310803 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:37.312542 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:37.423385 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:10:37.429155 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:10:37.475543 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240501 14:10:37.500311 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:37.504619 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:37.511790 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.513445 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:37.523965 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:37.535691 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:10:37.746482 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:37.750967 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:37.764818 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:37.781558 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:37.787758 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:37.788535 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:37.918637 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 14:10:37.971302 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240501 14:10:38.019688 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:38.022842 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:38.040485 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.040904 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.050304 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:38.256945 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:10:38.279363 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:38.280107 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.291229 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:38.305334 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:38.305536 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:38.319766 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:10:38.380095 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:38.398722 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:10:38.523680 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:38.531075 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.533280 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:38.536140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:38.546896 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:38.745864 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:38.750579 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240501 14:10:38.755124 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:38.769237 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:38.780121 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:38.781713 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:38.818140 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:39.010973 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:10:39.026958 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:39.035459 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:10:39.037290 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.043674 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:39.055573 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:39.073696 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:39.106344 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:10:39.195262 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:10:39.201311 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:10:39.238718 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:10:39.296413 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:39.302333 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:39.314563 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.317838 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.335690 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:39.340358 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:10:39.403107 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:39.552911 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:39.557742 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.558933 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:39.574187 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:39.591995 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:39.784526 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:10:39.790894 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:39.804862 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:39.812685 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:39.815623 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:39.835029 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:39.842712 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:39.968261 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240501 14:10:40.053920 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:40.055384 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:40.060815 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.111763 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:40.118871 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:40.178717 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:10:40.262447 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:40.298123 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:10:40.356379 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:40.366879 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:40.379696 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:40.383854 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240501 14:10:40.387502 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.393347 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:40.403232 6603 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:40.403417 6603 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:10:40.404340 6603 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:10:40.405977 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:40.406567 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:10:40.406858 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:10:40.417315 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:40.417469 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:40.418188 6603 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:40.656862 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:40.673759 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:40.678740 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:40.687570 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:40.689858 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:10:40.692792 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:40.718513 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:10:40.846386 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:40.907538 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:40.916110 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:40.918799 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:40.935727 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:10:40.939419 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:40.943328 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:41.096237 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240501 14:10:41.144800 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.147068 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.157377 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:41.165069 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.169019 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:41.208793 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:41.259352 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:41.326110 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:10:41.327117 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:10:41.397284 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.398334 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.402173 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.424302 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.446980 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:41.605479 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:10:41.636160 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.642983 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.654846 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:41.661586 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.663059 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:41.685494 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:10:41.730551 6603 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:41.730775 6603 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:10:41.731560 6603 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:10:41.732321 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:10:41.733007 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:10:41.734128 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:41.734298 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:10:41.735257 6603 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:41.744786 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:10:41.760880 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:41.820887 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:10:41.870352 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.887887 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:41.899345 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:41.902254 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:41.904740 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:10:41.920272 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:41.981096 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:10:42.141556 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.146996 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:42.148138 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:42.155439 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:42.176838 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:42.308900 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:10:42.335325 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:42.336207 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:42.418238 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240501 14:10:42.423149 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:42.441728 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:42.446671 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.451625 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:42.468545 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:42.556269 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:10:42.685356 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:42.694931 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:42.698396 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:42.707379 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:42.709820 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:42.808435 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:10:42.947786 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:42.949791 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:42.953610 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:42.976001 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:10:42.983668 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.003983 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:43.005607 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:10:43.006728 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 14:10:43.007547 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:43.072263 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:10:43.161638 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240501 14:10:43.198473 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.204360 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.219753 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.224610 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:43.249653 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:43.366540 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:43.434082 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.434463 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.435992 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.476570 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:43.496945 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:43.635002 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 14:10:43.689244 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.702348 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.712216 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.726521 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.730921 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:43.815645 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:43.824025 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:10:43.838183 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:43.862615 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240501 14:10:43.931231 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.944924 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:43.948097 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:43.950587 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:43.969910 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:44.152698 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:10:44.166918 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:44.175770 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:44.190855 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:44.195684 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.217595 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:44.277947 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:10:44.291205 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:10:44.296295 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:10:44.306897 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:10:44.432801 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:44.501309 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:44.501575 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:44.516019 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.522781 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:44.554448 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:44.766706 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:44.779019 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:44.784966 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:44.798549 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:44.809211 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:44.892714 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:44.998975 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240501 14:10:45.016923 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:45.024116 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:45.025184 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.062496 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.095957 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:45.141811 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:10:45.277833 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.285116 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:45.304394 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:45.323567 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.327654 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:45.336519 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:45.356096 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:10:45.363736 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:45.487109 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240501 14:10:45.569325 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:45.569801 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:10:45.578641 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.615737 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:45.635460 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:45.635738 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:45.689987 7749 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:10:45.690222 7749 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:10:45.691174 7749 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:10:45.693253 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:10:45.698896 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:10:45.699476 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:10:45.699903 7749 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:45.727284 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:10:45.826290 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:45.827207 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:45.829953 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:45.851819 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:45.869366 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:45.889286 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:46.095542 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:10:46.124310 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.124712 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.132174 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:46.134374 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:46.142952 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:46.240926 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:46.304059 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:46.304771 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:10:46.367933 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.369539 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.378430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:46.389205 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:46.390955 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:46.407070 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:46.408123 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:46.452616 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 14:10:46.550407 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:10:46.604735 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.620553 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.634932 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:46.635919 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:46.636896 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:46.704480 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:46.735792 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:10:46.801123 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 14:10:46.899260 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.899919 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:46.900787 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:46.912050 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:46.923795 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:46.926729 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:47.123836 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:10:47.164393 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.167207 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.178097 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:47.183140 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:47.198788 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:47.312124 6787 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:10:47.312394 6787 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:10:47.313177 6787 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:10:47.315752 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:10:47.315955 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:47.316040 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:47.316321 6787 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:47.370487 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:47.371690 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:47.441833 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.445292 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:47.464550 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.476491 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:47.476624 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:47.506199 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:47.604066 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240501 14:10:47.693528 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.696542 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.703490 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:47.722733 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:47.725545 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:47.787883 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:47.817710 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:47.828492 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:47.863850 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:10:47.910701 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240501 14:10:47.932082 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.933211 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:47.946086 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:47.947559 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:47.969041 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:48.210106 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.228199 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.229642 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:48.230777 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:48.253674 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:48.326778 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:48.366175 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:10:48.368384 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:10:48.394929 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:48.504673 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.508481 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:48.509353 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.522315 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.529304 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:48.635386 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:48.705011 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240501 14:10:48.731531 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.737907 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:48.743350 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.759910 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:48.766170 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:48.781214 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:48.787194 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:10:48.862262 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:48.969530 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:10:48.977814 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.979250 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:48.997467 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:49.005702 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:49.008771 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.226135 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:49.250301 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:49.257441 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:49.276389 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:49.289568 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:49.294992 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:49.295976 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:10:49.318508 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:49.417567 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:10:49.427656 6603 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:49.427886 6603 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:10:49.428807 6603 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:10:49.430470 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:49.438393 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:10:49.438823 30927 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:10:49.439646 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:49.439772 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:49.440143 6603 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:49.520303 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:49.525506 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:49.529594 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:49.537998 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:49.549620 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:49.565014 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:49.675578 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:10:49.800683 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:49.815176 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:49.821779 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:49.838407 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:49.842960 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:49.853230 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:50.090479 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:50.097467 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:50.102376 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.104179 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:50.111196 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:50.237090 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:50.253870 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240501 14:10:50.332598 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:50.353713 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:50.354823 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:50.382524 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:50.387020 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.394752 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:50.524394 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:10:50.542866 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240501 14:10:50.608858 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:50.614811 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:50.640264 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:50.643903 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.646036 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:50.756788 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:10:50.849051 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:50.850708 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:50.865609 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:50.871405 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:50.886399 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:50.990538 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:51.063648 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 14:10:51.098376 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.102818 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:51.104990 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.132593 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.134537 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:51.207885 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:10:51.209580 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:10:51.244122 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:10:51.306979 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:10:51.337863 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.339865 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.368556 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:51.392426 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:51.398830 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:51.402073 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:51.472648 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:10:51.498057 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:10:51.502422 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:10:51.569396 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 14:10:51.642180 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.654373 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.665941 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:51.676854 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:51.678965 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:51.719172 28940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 47754us
W20240501 14:10:51.752920 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:51.779995 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:51.928731 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.932206 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:51.941244 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:51.960842 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:51.961012 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:51.977342 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:10:52.091090 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:10:52.135247 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:10:52.205437 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.207077 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.208215 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.211143 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:52.239884 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:52.271078 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:52.437014 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.441671 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.457230 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:52.478673 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.491183 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:52.558255 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:10:52.579777 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:10:52.671999 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.680757 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:52.682086 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.696460 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.708092 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:52.774969 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240501 14:10:52.851737 8664 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:52.851991 8664 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:10:52.852789 8664 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:10:52.853418 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:10:52.854727 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:52.859516 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:52.859658 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:10:52.859892 8664 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:52.866930 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:52.939553 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.948045 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:52.966365 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:52.966923 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:52.974395 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:53.032150 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:10:53.052867 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 14:10:53.119915 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:10:53.208153 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:53.213388 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:53.220021 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:53.228660 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.233453 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:53.435482 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:53.448835 9179 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:10:53.449038 9179 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:10:53.449793 9179 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:10:53.452008 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:10:53.452796 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:10:53.452922 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:10:53.453449 9179 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:53.512710 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:53.513471 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:53.526525 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:53.535786 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:53.551199 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:53.651300 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:10:53.657040 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:10:53.658170 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:53.671217 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:10:53.698570 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 14:10:53.751304 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:53.761937 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:53.764832 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:53.776108 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:53.796186 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:53.991837 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:10:53.992651 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:53.993249 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:53.993592 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:54.004904 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:54.047293 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:54.050221 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:54.063514 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:10:54.129591 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:10:54.235893 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:54.237799 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:10:54.251123 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:54.260778 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:54.265430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:54.273350 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:54.425627 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:10:54.447551 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:54.510098 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:54.518003 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:54.523267 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:54.532897 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:54.542933 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:54.654894 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:10:54.688366 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:10:54.771169 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:54.774168 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:54.778538 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:54.789642 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:54.805622 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:54.954340 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:10:55.010308 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:55.016007 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.033392 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.037714 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:55.056106 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.057176 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:55.305160 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.305637 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.313874 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:55.331116 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 14:10:55.336205 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:55.354301 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:55.518225 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 14:10:55.565183 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:10:55.566228 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:10:55.566459 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:55.571955 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.577251 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.578047 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.605911 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:55.609511 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:55.796149 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.810145 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:55.817844 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:55.820574 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:55.830406 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:55.852808 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:55.930994 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240501 14:10:56.063545 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:56.076156 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:56.081858 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:56.091879 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:56.103051 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:56.103526 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:56.265352 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:10:56.274519 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:56.275645 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:56.276489 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:56.277958 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:10:56.283818 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:56.299403 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:56.308380 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:56.309841 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:56.318199 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:10:56.513082 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:10:56.545957 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:56.568747 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:56.576571 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:56.580646 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:56.589617 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:56.629892 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:56.648377 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:56.688911 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:56.799010 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:56.805529 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:56.813059 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:56.817299 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:56.818042 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:56.867192 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:10:57.029053 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:10:57.056030 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:57.088944 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.095227 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.098469 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:57.112267 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:57.126246 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:57.190564 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:57.320667 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.324448 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:57.343885 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.355487 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.375412 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:57.458956 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:57.533056 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:10:57.584556 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:10:57.603140 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:57.610867 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.620271 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.642474 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:57.645232 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:57.694023 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 14:10:57.803743 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240501 14:10:57.832020 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.838012 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:57.840490 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:57.879292 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:57.904959 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:57.956948 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:58.000883 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 14:10:58.068440 9244 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:10:58.068641 9244 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:10:58.069628 9244 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
W20240501 14:10:58.070611 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:58.072018 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:58.071993 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:10:58.072651 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:10:58.072784 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:10:58.073272 9244 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:58.086340 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.095273 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:58.099862 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.111111 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.111217 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:58.150708 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:58.315737 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.342332 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:58.352933 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.365430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:58.379141 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:58.449631 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 14:10:58.474084 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:10:58.548584 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:10:58.554216 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:58.660566 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.664687 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.679034 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:58.688174 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.718217 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:58.876438 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:10:58.877477 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:10:58.881245 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:58.929560 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.933487 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:58.936002 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:58.955299 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:58.961064 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:58.974963 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:10:59.145325 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:10:59.161219 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:10:59.186801 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:59.195336 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:10:59.197803 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:59.215363 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:59.261078 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:59.266546 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:10:59.303954 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:59.434572 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:10:59.440917 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:10:59.441459 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:59.450850 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:59.469416 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:59.475380 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:59.649488 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:10:59.669926 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:59.676955 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:59.691680 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:59.692747 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:10:59.693305 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:10:59.891572 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:10:59.909170 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:59.913969 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:10:59.922933 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:10:59.924054 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:10:59.928930 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:10:59.940924 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:00.125536 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:00.133977 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:00.146184 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:00.150508 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:00.153146 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:00.191255 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:11:00.192588 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240501 14:11:00.430847 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:00.433346 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:00.450392 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:00.457842 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:00.465330 9244 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:00.465526 9244 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:11:00.466358 9244 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
W20240501 14:11:00.468890 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:00.469316 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:11:00.470089 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:11:00.470217 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:11:00.470767 9244 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:00.471196 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240501 14:11:00.472070 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:00.499424 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:00.676470 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:11:00.683176 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:00.692193 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:00.695806 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:00.699307 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:11:00.703207 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:00.750552 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:00.840559 10281 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:11:00.840766 10281 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:11:00.841518 10281 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:11:00.843266 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:11:00.843542 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:11:00.843647 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:11:00.843901 10281 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:00.889915 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:00.893822 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240501 14:11:00.936793 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:00.937192 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:00.941148 6787 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:11:00.941366 6787 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:11:00.942119 6787 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:11:00.949975 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:00.949990 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:00.951596 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:00.951798 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:00.951893 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:00.954967 6787 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:00.976356 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:01.005805 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:11:01.237336 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:11:01.238867 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:01.257395 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:01.260476 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:01.270278 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:01.274477 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:01.297138 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:11:01.298517 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:01.317065 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:11:01.482663 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:11:01.518122 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:01.528893 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:01.531790 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:01.544392 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:01.553951 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:01.563042 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:01.575683 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:01.766546 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:01.771363 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:01.784843 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:01.798964 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:01.800201 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:01.861584 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:01.906374 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:11:01.982725 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:11:02.000286 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:02.018026 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:02.020079 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:02.034701 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:02.045064 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:02.055399 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:02.061758 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:02.294137 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:02.295853 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:02.313377 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:02.315929 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.335009 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:02.437496 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:02.453773 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:11:02.586068 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:02.604712 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:02.623198 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:02.638536 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:02.640161 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:11:02.643455 6787 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:11:02.643631 6787 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:11:02.644364 6787 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:11:02.653743 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:02.660715 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:02.661063 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:11:02.661181 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:02.661432 6787 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:02.752243 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:11:02.854172 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:02.859071 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:02.871755 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:02.892658 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:02.896064 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:02.902348 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:02.970530 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:11:03.003580 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 14:11:03.081548 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:11:03.105880 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 14:11:03.106877 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.120808 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.127849 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:03.141953 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.161087 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:03.166803 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 14:11:03.322996 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:03.395327 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.403368 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.407807 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:03.428674 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.430819 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:03.509748 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:11:03.573035 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:11:03.663960 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.664696 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.677999 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.680887 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:03.699364 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:03.702134 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:11:03.718417 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:11:03.731784 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:11:03.867295 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:03.878842 9244 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:03.879070 9244 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:11:03.880163 9244 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:11:03.881273 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:11:03.881536 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:11:03.882457 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:11:03.884219 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:03.884353 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:03.884613 9244 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:03.889991 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:03.926721 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.938117 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:03.938939 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:03.939906 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:03.968662 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:04.187906 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:11:04.190963 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.201087 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.202433 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:04.218689 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240501 14:11:04.219117 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.224823 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:04.281625 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:04.384090 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:11:04.452607 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.465812 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:04.471400 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.485536 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.487969 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:04.523851 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:11:04.593001 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:11:04.684907 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:04.709332 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.717487 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:04.723574 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:04.729022 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.743481 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:04.745034 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:04.751006 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:04.966691 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.970127 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:04.981706 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:04.983036 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:04.987339 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:05.199008 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:05.210382 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:05.231211 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:05.237654 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 14:11:05.243247 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:05.255162 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:05.265900 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.465785 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:05.468406 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:05.471241 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:05.489024 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:05.489879 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 14:11:05.493360 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:05.669387 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:11:05.675709 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:05.726576 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 14:11:05.757741 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:05.764755 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:05.765199 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:05.775130 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:05.781819 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:05.883179 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:11:05.998234 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 14:11:06.034996 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.044903 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.050014 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:06.082235 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.085541 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:06.097797 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:11:06.104979 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:11:06.161259 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:11:06.262224 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:06.267915 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.271697 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:06.288944 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.298538 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.299798 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:06.320708 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:06.492647 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:11:06.516824 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.527143 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:06.532550 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.547289 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:06.547906 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:06.551509 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:06.767095 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:06.783066 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.795456 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:06.795912 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:06.804106 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:06.808310 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:06.819147 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:06.992637 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:07.030081 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.044766 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.052945 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:07.060683 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.065697 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:07.067469 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:07.074481 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:11:07.143159 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:11:07.199468 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:07.275050 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.279296 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.279675 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:07.289682 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:07.309294 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:07.313022 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:11:07.548218 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:07.548295 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.563488 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.569475 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:07.588090 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:07.641132 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:11:07.707298 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:07.760679 9244 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:07.760941 9244 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:11:07.761757 9244 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:11:07.763533 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:11:07.773046 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:07.773274 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:07.773620 9244 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:07.788007 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.793169 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:07.800140 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:07.802788 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:07.809206 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:07.904035 6787 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:07.904273 6787 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:11:07.915757 6787 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:11:07.917668 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:11:07.918009 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:11:07.919046 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:07.920711 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:07.924643 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:07.924831 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:07.925742 6787 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:08.061434 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240501 14:11:08.061928 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:08.078147 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:08.080919 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.085530 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:08.101760 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:08.103139 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:11:08.180342 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 14:11:08.259663 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:11:08.303260 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:08.321661 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:08.322244 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:08.359405 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:08.376439 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:08.377403 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:08.381467 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:08.456326 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:11:08.458074 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:11:08.464823 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:11:08.596438 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:08.597867 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:08.609123 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:08.623826 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:08.655191 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:08.743502 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:08.747745 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:11:08.823719 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:08.831851 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:08.845507 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:08.845638 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:08.871906 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:08.885777 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:09.080636 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:09.083114 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:09.086494 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:09.091949 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:09.107520 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:09.183228 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:11:09.334451 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:09.340924 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:09.344189 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240501 14:11:09.359112 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:09.366431 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:09.375042 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:09.395641 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.409864 11502 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:11:09.410068 11502 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:11:09.410967 11502 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:11:09.412986 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:09.414315 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:09.414418 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:11:09.415295 11706 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:09.427345 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:09.429967 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:09.460853 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:11:09.555199 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:09.622179 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:11:09.623572 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:09.627194 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:09.640590 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:09.658226 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:09.678862 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:09.777290 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:09.862134 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:09.863700 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:09.870112 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:09.884199 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:09.913199 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:10.134938 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:10.147931 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:10.159974 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:10.175081 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.176757 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:10.278766 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:11:10.291571 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 14:11:10.317802 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:10.416841 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:10.429709 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:10.438751 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:10.464733 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:11:10.468773 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:10.475167 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:10.609019 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240501 14:11:10.669617 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:10.680593 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:10.686955 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:10.693194 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:10.696854 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:10.746326 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:10.756482 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:10.821018 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:10.908386 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:10.909797 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:10.912302 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:10.913210 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 14:11:10.922808 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:10.923170 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:11.051721 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:11:11.136063 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:11.139269 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:11.140130 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:11.172446 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.187503 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:11.333256 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:11:11.392043 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:11.398370 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 14:11:11.415263 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:11.418430 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:11.421408 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:11.428539 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:11.429075 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:11.442663 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:11.644157 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:11.660173 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:11.664856 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:11.671272 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.684203 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:11.703641 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:11:11.796718 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:11.901958 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:11.907176 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:11.916582 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:11.926092 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:11.928289 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:11.987614 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:12.010735 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:12.047012 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:12.051673 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:12.160226 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:11:12.161944 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:12.163015 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:12.163839 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:11:12.175532 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:12.180382 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.198218 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:12.399947 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:12.400749 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:12.412925 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:12.438601 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.519927 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:12.533820 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:12.670377 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:12.672592 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:12.675421 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:12.697503 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:12.701262 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:12.738766 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:12.864815 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 14:11:12.885448 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240501 14:11:12.955636 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:12.961796 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:12.962211 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:12.970852 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:12.977146 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:13.056192 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:13.057150 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:13.140815 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:11:13.177300 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.179368 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.188236 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:13.210609 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:11:13.233003 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240501 14:11:13.236478 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.236871 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:13.377197 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:11:13.401116 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 14:11:13.437317 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.441429 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.458820 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:13.461058 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 14:11:13.467917 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:13.476753 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:13.575887 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:13.679086 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.680172 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.694308 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:13.695401 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:13.731428 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:13.876217 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:13.937983 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.938021 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:13.948210 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:13.955991 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:13.956081 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:13.956879 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:13.968956 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:14.154289 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:14.178321 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:14.178617 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:14.194478 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:14.209482 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.215479 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:14.233719 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:14.422122 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:14.426383 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:14.430059 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:14.450114 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.452529 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:14.457201 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:14.511778 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:11:14.559132 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:14.669122 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:14.684718 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:14.685132 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:14.699800 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:14.729882 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:14.731668 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:14.769969 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:14.797361 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:14.954685 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:14.954775 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:14.955097 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:14.956166 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:14.964744 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:14.983250 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:15.211844 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:15.217279 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:15.217777 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:15.218492 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.240432 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:15.307613 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:11:15.324278 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:15.466452 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:15.478368 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:15.480847 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:15.481664 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:15.510191 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:15.511899 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:15.597328 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240501 14:11:15.723654 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:15.733510 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:15.739022 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:15.743934 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:15.744796 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:15.748178 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:11:15.755086 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:11:15.768723 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240501 14:11:15.774749 12645 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:15.774943 12645 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:11:15.776026 12645 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:11:15.776202 12645 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:15.776345 12645 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:11:15.777258 12645 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:11:15.778728 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:11:15.779003 30927 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:11:15.783005 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:15.783136 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:15.783258 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:11:15.783512 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:15.783856 12645 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:15.799082 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:11:15.799986 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:11:15.800236 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:11:15.800748 12729 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:15.918121 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:15.983436 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:15.984586 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:15.993108 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:15.998699 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.022817 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:16.059531 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:11:16.075095 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:16.221701 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:16.224879 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:16.241648 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:16.246352 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:16.251648 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:16.271574 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:16.401587 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:16.414578 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:11:16.465116 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:16.518034 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:16.524322 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:16.524534 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:16.525512 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:16.534893 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.555632 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:16.616474 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:16.746783 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:16.747725 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:16.768469 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:16.775835 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:16.779567 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:16.877235 6787 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:11:16.877503 6787 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:11:16.878461 6787 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:11:16.879838 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:16.883528 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:16.883767 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:16.883878 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:16.884196 6787 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:17.007983 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.033310 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:17.048723 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:11:17.070899 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:17.072942 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.073472 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:17.127799 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:17.267372 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 14:11:17.276531 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.287179 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.291209 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:17.299957 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:17.311149 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:17.415895 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:17.478125 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:17.479296 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:17.534577 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:17.537521 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.545374 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.564289 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:17.568526 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:17.712090 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 14:11:17.760939 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.773993 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:17.782332 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:17.789994 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:17.791421 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:17.816361 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:17.920569 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:18.005443 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.018599 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:18.021513 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.023363 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.055604 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:18.104669 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:11:18.133329 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 14:11:18.145911 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:18.163242 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240501 14:11:18.264458 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.269433 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:18.279201 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.293937 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:18.295305 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:18.329679 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:18.400655 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:11:18.455605 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240501 14:11:18.523944 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.531546 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.542384 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:18.554894 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:18.564407 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:18.570714 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:18.591461 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:11:18.684650 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:11:18.741195 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:18.777554 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.784586 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:18.785001 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:18.801803 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:18.820982 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:18.947587 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:19.029085 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:19.030284 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:19.045208 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:19.059859 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:19.065625 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:19.073119 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:19.108798 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:11:19.253849 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:19.289300 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:19.291183 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:19.299161 13260 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:19.299355 13260 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:11:19.300130 13260 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:11:19.310755 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:19.313906 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:19.325317 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:19.325580 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:19.325881 13260 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:19.330826 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.333513 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:19.579644 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:19.586264 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:19.588766 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:19.601269 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:19.610697 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:19.623276 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:19.641415 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:19.694042 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:19.749166 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:19.816131 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:19.832587 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:19.846133 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:19.848605 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:19.884321 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:11:19.887135 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:20.010898 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:20.117815 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.138947 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.142297 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:20.153805 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:20.155622 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:20.183723 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:20.249868 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 14:11:20.292654 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:20.392828 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.405267 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:20.411782 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:20.415122 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.431424 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:20.639019 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:11:20.650931 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 14:11:20.654476 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.662514 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.672138 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:20.683429 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:20.685477 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:20.689217 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 14:11:20.709326 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:20.803393 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:20.892166 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.905315 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:20.905704 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:20.907419 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:20.936455 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:20.985136 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:21.104954 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.113338 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:21.121405 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.121999 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.126787 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:21.141830 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:11:21.183718 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:21.335738 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.349895 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.362309 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:21.377504 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:11:21.405508 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:21.408540 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:11:21.410542 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:21.410939 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:21.437052 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:21.584496 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:11:21.597370 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:21.619967 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.628021 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.642256 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:21.671936 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.687176 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:21.913072 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.915499 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:21.916591 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:21.917426 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:21.920753 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:22.024597 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:22.046265 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:22.051004 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:11:22.164000 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:22.169231 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.186606 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:22.196135 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.211560 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.219519 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:22.425539 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.434917 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:22.437044 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.467777 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:22.472638 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:22.522253 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 14:11:22.528752 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:22.581436 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:22.682057 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.684537 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.689800 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:22.694099 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:22.701395 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:22.767897 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:22.925632 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.933123 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:22.936528 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:22.949991 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:22.953568 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:22.985039 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:11:23.006145 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:11:23.023555 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:23.046372 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 14:11:23.133023 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 14:11:23.138479 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:11:23.163332 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:23.175091 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:23.186605 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.187678 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:23.188134 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:23.440425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:23.447386 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:23.448488 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:23.456027 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:11:23.464777 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:23.468391 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:23.469993 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 14:11:23.470867 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:23.519982 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:23.696287 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:23.701001 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:23.703563 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:23.703539 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:23.729620 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:23.926985 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:11:23.963063 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:23.963229 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:23.964361 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:23.980280 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:23.985015 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:24.010087 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:24.012143 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:24.044008 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:11:24.048223 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:24.069589 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:24.125867 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 14:11:24.127400 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fb04ec77da194832aa63d69aba307de3: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:24.235494 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:11:24.246731 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:24.263664 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:24.275295 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:24.280222 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.295370 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:24.457206 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:11:24.523159 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:24.526863 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:24.531620 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:24.551862 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:24.552480 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:24.571295 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.767372 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:24.772006 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:24.784752 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:24.785894 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:24.799881 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:24.827595 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 14:11:24.892334 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:11:24.943506 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:24.993662 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:24.999749 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:25.002875 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:25.027269 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:25.034104 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:25.093488 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:25.100976 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:11:25.231431 13447 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:25.231645 13447 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:11:25.232497 13447 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:11:25.234045 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:11:25.235018 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:11:25.235257 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:25.235385 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:11:25.235950 13447 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:25.277673 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:25.279402 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:25.289053 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:25.310909 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:25.344882 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:25.345516 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:25.375253 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:25.386741 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:25.508049 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240501 14:11:25.527514 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:25.535166 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:25.541468 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:25.546931 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:25.570322 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:25.688894 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240501 14:11:25.776211 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:25.787434 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:25.791020 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 14:11:25.791992 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:25.805810 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:25.806715 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:25.868796 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:11:25.988562 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:26.015826 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:26.023983 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:11:26.029110 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:26.032662 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:26.061745 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:26.073927 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 51379.9us
I20240501 14:11:26.120622 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:26.270519 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:26.275259 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:26.289297 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:26.311391 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:26.315454 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:26.381822 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:26.405591 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:26.447000 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 14:11:26.447876 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:11:26.464218 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:11:26.526180 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:26.537292 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:26.544939 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:26.546306 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:26.552508 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:26.555552 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:26.688697 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:11:26.776588 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:26.783772 14050 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:11:26.783970 14050 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:11:26.784870 14050 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
I20240501 14:11:26.786111 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:26.787395 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:26.791627 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:26.807678 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:26.812470 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:11:26.812669 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:11:26.812933 14050 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:26.826181 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:26.950191 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:11:26.981634 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:11:27.030938 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:27.073449 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.079902 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:27.085616 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.092993 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:27.104226 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.293382 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:27.294173 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.297227 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.319682 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.327852 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:27.350948 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240501 14:11:27.528378 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.536914 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.539219 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:27.565850 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.566290 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:27.572851 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:27.587188 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:27.592315 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:27.657418 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:11:27.764278 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.793891 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:27.801247 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:27.802335 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:27.806181 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:27.892117 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:11:28.027782 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.041141 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:28.046993 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:28.055152 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.056833 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.081337 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:28.092674 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 14:11:28.100582 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 14:11:28.121198 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 14:11:28.277890 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.280041 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.286001 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:28.326249 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:28.366854 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:11:28.376636 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:28.440063 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:28.474010 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:28.512558 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:11:28.600574 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.602380 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.605161 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.625878 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:28.626173 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:28.654935 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 14:11:28.849635 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.849823 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:28.850405 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:28.875968 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:28.884491 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:28.927400 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:11:28.936879 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:11:28.977418 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:11:28.982874 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:28.987041 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:11:28.998996 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:11:29.096365 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.114917 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.117282 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.119378 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:29.130084 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:29.279261 13447 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:29.279465 13447 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:11:29.280529 13447 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:11:29.284773 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:29.294955 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:11:29.295272 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:11:29.298652 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:29.298806 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:29.299110 13447 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:29.344321 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.346429 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.368369 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:29.382864 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:29.400682 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:29.406530 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:29.521330 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:11:29.563412 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:29.625708 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.626356 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:29.640974 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.651410 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:29.663057 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.844661 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.854897 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:29.872628 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:29.883939 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:29.909050 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:29.945878 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:11:29.946749 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:30.024494 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:11:30.071588 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240501 14:11:30.105306 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.112166 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.117928 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:30.122440 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.135560 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:30.198009 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:11:30.333282 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.339666 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.348069 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:30.376731 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:30.390904 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:30.391171 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:30.488185 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 14:11:30.573714 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:11:30.574995 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:11:30.627269 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 14:11:30.668633 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.674862 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.684818 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:30.709167 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:30.735947 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:30.758332 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240501 14:11:30.958572 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.960546 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:30.960652 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:30.968647 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:30.984297 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:30.998498 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:31.113782 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:11:31.204728 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:31.204927 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.210733 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:31.214058 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:31.215596 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:31.303100 13631 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:11:31.303313 13631 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:11:31.304009 13631 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:11:31.306099 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:31.306293 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:31.306396 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:31.306720 13631 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:31.353331 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:31.446022 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:31.446089 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:11:31.446491 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:31.463862 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:31.489352 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:31.491022 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:31.504303 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:11:31.510232 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:31.513774 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:11:31.580636 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:31.640857 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:31.719559 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:31.738032 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:31.755127 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:31.777781 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:31.778915 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:31.820883 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:31.906968 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:32.000300 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.005009 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:32.007076 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.025476 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.040304 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:32.121757 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:32.199568 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:11:32.201089 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e0bf08cbb6d347d783b1fa7762c802df: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:32.243489 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.246851 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.260933 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:32.268177 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.277630 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:32.470870 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:32.514544 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:32.517678 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.519953 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.520282 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:32.520992 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:32.548099 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:11:32.570422 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:11:32.597231 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:11:32.653649 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:32.747541 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.762862 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:32.775210 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:32.779727 13631 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:32.779918 13631 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:11:32.781013 13631 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:11:32.782560 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
W20240501 14:11:32.788451 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:32.795498 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:32.836782 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:32.837410 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:32.837783 13631 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:32.860774 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:32.880163 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:11:32.883037 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:11:32.887734 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:32.970258 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240501 14:11:33.030112 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:33.035666 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:33.046859 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:33.056489 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.069201 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:33.157583 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:11:33.163267 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:33.278687 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:33.281821 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:33.282786 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:33.290752 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240501 14:11:33.311180 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:33.326509 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:33.544703 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:11:33.548231 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:11:33.567584 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:33.576913 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:33.600005 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:33.617535 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:33.618834 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:33.622951 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:33.753507 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:11:33.852356 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:33.859284 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:33.864846 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:33.871380 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:33.894593 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:33.908227 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:33.963801 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:11:34.088160 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:34.123485 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:34.127995 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:34.142072 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.144066 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:34.146270 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:34.153811 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:34.163700 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:11:34.188937 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:34.221485 15349 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:34.221707 15349 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:11:34.222488 15349 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:11:34.224222 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:34.226094 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:34.226241 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:34.226536 15639 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:34.249696 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:11:34.391844 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:34.392323 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:34.411613 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:11:34.416756 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.416906 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:34.417799 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:34.662489 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:34.669209 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:34.669989 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:34.679710 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:34.691205 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:34.707034 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:34.725257 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:11:34.906967 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:34.910889 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:34.917272 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:34.931500 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:34.941771 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:35.036962 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 14:11:35.048486 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:11:35.124989 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.132016 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.137406 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:35.142985 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.166061 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:35.241096 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:35.254139 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:11:35.259632 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:11:35.322899 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240501 14:11:35.395488 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:35.395624 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.401036 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.405727 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:35.406082 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.627326 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.630949 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.652640 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:35.667403 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:35.671577 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:35.697175 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:35.802830 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 14:11:35.887187 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.893551 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:35.903976 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:35.938880 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:35.955212 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:36.002329 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 14:11:36.053292 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:36.064560 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 14:11:36.116819 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:36.139056 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:36.140859 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:36.148191 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:36.154903 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:36.164646 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:36.366031 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:36.376305 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:36.395714 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:36.400796 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:36.417322 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:36.426254 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:36.438549 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:36.445910 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:11:36.498476 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:11:36.585956 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:11:36.591463 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:36.632872 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:36.638254 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:36.642053 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:36.644454 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:36.648970 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:36.722595 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:36.821172 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:36.863864 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:36.894898 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:36.902415 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:36.917719 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:36.918295 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:36.939581 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:37.076118 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:37.137667 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.147080 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.147363 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:37.161417 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:37.175201 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:37.214128 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:37.343300 16078 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:11:37.343511 16078 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:11:37.344331 16078 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:11:37.347036 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:37.367967 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:37.368172 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:11:37.368553 16078 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:37.390580 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.404978 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:37.409215 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.426719 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.513263 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:37.569702 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:11:37.686689 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240501 14:11:37.688064 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.705464 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.713603 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.718066 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:37.724166 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:37.730918 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:37.787770 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:37.887568 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:37.889459 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:11:37.911600 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240501 14:11:37.936830 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.949910 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:37.950541 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:37.954560 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:37.959785 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:38.179247 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:38.180330 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:38.200973 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:38.201242 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.213886 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:38.215620 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:38.303969 16188 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:38.304214 16188 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:11:38.305087 16188 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:11:38.305392 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:11:38.305635 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:11:38.314742 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:38.322961 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:38.323124 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:38.323405 16188 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:38.397210 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:38.440395 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:38.443933 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:11:38.450168 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:38.456840 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:38.464365 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240501 14:11:38.485831 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.499555 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:38.563683 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:38.566013 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:38.566962 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:38.720968 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:38.722014 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:38.728878 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:38.735859 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:38.746790 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:38.748056 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:38.876509 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:38.959602 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:38.961716 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:38.962881 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:38.979960 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:38.994520 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:39.046597 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:39.106974 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:11:39.109800 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:11:39.137766 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:39.199698 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:39.214855 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:39.218020 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:39.228314 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.236083 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:39.244293 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:39.287883 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:39.489441 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:39.500696 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:39.516320 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:39.519412 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:39.525743 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:39.528116 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:39.555851 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:39.777558 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:39.780835 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:39.791684 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:39.796363 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:39.816919 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:39.934993 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:40.029624 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.042860 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:40.045150 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.063338 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.069413 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:40.156461 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 14:11:40.217347 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:40.275911 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.279953 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.298873 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:40.303977 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:40.304414 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:40.315495 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:40.459892 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:40.528071 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:40.529309 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.530387 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.540714 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:40.558490 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:40.562685 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.564049 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:40.566967 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:11:40.613138 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240501 14:11:40.809270 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.809726 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:40.817625 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:40.821210 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:40.821802 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:41.009415 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:41.055740 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:41.065287 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:41.067870 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:41.085357 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.102645 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:41.104532 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 14:11:41.120296 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:11:41.226960 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:11:41.258390 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:41.322723 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:41.329391 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:41.347819 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:41.356261 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.364815 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:41.413329 16703 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:41.413586 16703 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:11:41.414589 16703 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:11:41.421494 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:41.421803 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:41.422778 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:41.422941 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:41.426589 16703 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:41.429447 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:41.604166 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:41.608867 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 14:11:41.614301 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:41.616276 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 14:11:41.641744 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:41.648787 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.653594 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:41.654897 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:41.819070 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:41.847218 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:41.863909 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:41.865907 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:41.869871 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:41.878464 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:41.884403 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:42.007355 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:42.109186 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:42.126145 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:42.135753 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.137681 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:42.145273 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:42.146335 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:42.363555 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:42.386783 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:42.390134 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:42.418241 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:42.437546 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.451692 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:42.635910 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240501 14:11:42.641799 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:42.644202 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:42.645059 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:42.664410 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.693933 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:42.780375 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:11:42.874871 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:11:42.885774 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 14:11:42.887037 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:42.904088 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:42.905056 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:42.909803 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:42.915256 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:42.977981 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:42.982726 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:11:43.016392 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:43.089304 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:43.101311 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240501 14:11:43.181762 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:43.185071 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:43.200592 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.203629 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:43.213838 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:43.356256 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240501 14:11:43.421499 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:43.425017 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:43.440369 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:43.441784 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:43.443889 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.465839 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:43.545297 16703 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:43.545502 16703 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:11:43.546371 16703 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:11:43.546947 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:11:43.547632 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:11:43.547883 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:43.548018 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:11:43.548333 16703 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:43.581241 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:11:43.653914 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:43.668360 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:11:43.671126 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.674728 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:43.676604 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:43.678360 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:43.703945 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:43.740154 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:11:43.770386 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:43.863428 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:11:43.864719 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:43.923041 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:43.931267 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:43.931746 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:43.964263 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:43.965981 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:44.145318 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:11:44.170239 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.181865 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.182484 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:44.205300 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.219185 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:44.340672 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:44.348595 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:44.400560 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:44.427512 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.442170 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:44.455660 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.465591 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:44.468703 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:44.702582 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.704221 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.724653 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:44.726248 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:44.732023 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:44.758976 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:44.921403 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:44.941601 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.947839 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:44.958837 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:44.964022 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:44.988267 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:45.030728 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 74700.1us
I20240501 14:11:45.224872 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:45.226953 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:45.232055 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:45.243423 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:45.250397 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:45.253433 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:45.280902 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 14:11:45.296859 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:45.387594 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:11:45.500197 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:45.503939 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:45.525429 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:45.526324 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:45.538725 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:45.573825 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:11:45.584332 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:45.652370 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 14:11:45.674155 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:11:45.761051 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:45.761201 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:45.774957 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:45.779707 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:45.794104 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:45.833315 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 14:11:45.867045 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:46.009387 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:46.013031 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:46.024199 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:46.040439 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:46.044029 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:11:46.045753 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.249537 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:46.270362 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.270704 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:46.282094 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:46.286741 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:46.286908 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:46.318395 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:11:46.501145 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:11:46.526182 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:46.526556 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:46.533874 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:11:46.534997 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:46.540800 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:46.561751 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:46.612176 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 14:11:46.635277 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:46.751875 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:11:46.786195 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:46.787264 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:46.790685 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:46.799296 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:46.808859 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:46.814807 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:46.888727 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:11:46.938457 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:11:47.019543 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.020565 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.061792 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:47.066417 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:47.082258 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:47.156117 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:11:47.289875 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.298844 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.309836 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:47.314827 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:47.323669 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:47.332473 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:47.552758 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.558612 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.562013 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:47.590358 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:47.597752 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:47.726253 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:47.739358 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:11:47.745276 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 14:11:47.783350 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 14:11:47.791682 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.792943 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:47.797492 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:47.815053 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:47.826578 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:47.859992 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:47.976202 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:11:48.031008 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.044579 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.049499 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:48.050498 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:48.067032 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.123167 17346 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:11:48.123401 17346 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:11:48.124262 17346 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:11:48.126262 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:11:48.126497 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:48.126613 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:48.126961 17346 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:48.160120 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:48.170202 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 14:11:48.182492 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:48.210325 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:11:48.250401 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
I20240501 14:11:48.254863 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.263068 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:48.271140 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.276784 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:48.280772 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:48.441272 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:11:48.519511 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.530000 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:48.539228 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.547565 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:48.559270 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:48.605464 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:11:48.731627 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:48.732688 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:48.755633 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.757537 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:48.765377 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.779529 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:48.794885 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:11:48.810024 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:48.935055 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:11:48.988286 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:48.994280 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:49.014768 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:49.014827 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:49.038759 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:49.224402 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:49.239154 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:49.247372 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:49.253371 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:49.259323 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 14:11:49.265753 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.269032 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:49.336437 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:11:49.429713 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:49.439953 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:11:49.506446 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:49.517843 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:49.531077 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:49.537129 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:49.539582 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:49.718739 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:11:49.768620 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:49.777236 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:49.779137 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:49.790858 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:49.800766 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:49.802312 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.027659 18050 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:11:50.027869 18050 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:11:50.028815 18050 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:11:50.035943 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:11:50.039721 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:50.055115 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:50.055367 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:11:50.055680 18050 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:50.061671 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:50.062017 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.064939 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:50.068442 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:50.137694 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:11:50.176797 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:50.191162 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:11:50.302065 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:50.304472 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:50.307322 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:50.317725 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.327013 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:50.338902 16703 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:50.339144 16703 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:11:50.339953 16703 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:11:50.341547 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:11:50.342336 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:11:50.343892 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:50.344024 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:11:50.344796 16703 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:50.351140 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:11:50.376338 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:11:50.389549 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 14:11:50.448660 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240501 14:11:50.575294 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:50.582194 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:50.616719 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.620504 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:50.640461 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:50.705528 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 14:11:50.735561 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:50.814432 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:11:50.837234 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:50.840950 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:50.863688 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:50.878974 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:50.912029 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:50.945672 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:50.959303 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:51.017019 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:11:51.097971 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.100814 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.110646 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:51.128257 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.142715 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:51.253096 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:51.367357 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.389199 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.393421 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.396827 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:51.404549 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:51.636008 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.638173 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:51.643625 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:51.644259 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.656366 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:51.704562 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 14:11:51.807508 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:11:51.821969 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:51.842891 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:11:51.852746 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:51.910181 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:11:51.912552 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 14:11:51.922516 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.930899 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:51.936569 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:51.954308 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:51.962121 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.171124 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.176389 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.187309 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:52.190862 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:52.191568 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:52.222247 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:52.391767 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:52.416911 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.422016 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.442240 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.446487 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:52.452690 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:52.554987 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:52.611049 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:11:52.662042 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:52.664525 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.681692 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.691946 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:52.699224 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:52.745317 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:11:52.746763 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 14:11:52.872994 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240501 14:11:52.901733 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.911634 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:52.913378 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:52.926105 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:52.926528 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:52.927399 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:11:52.944224 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:53.061383 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240501 14:11:53.138888 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.157032 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.158519 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:53.162922 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:53.168471 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:53.173663 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 44797.1us
W20240501 14:11:53.261821 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:53.262729 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:11:53.401595 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.407841 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.414649 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:53.420089 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:53.439677 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:53.504768 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:11:53.526326 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:53.557663 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:11:53.631912 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:11:53.704839 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.706872 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:53.719295 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.720274 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:53.741672 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:53.782009 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:53.903116 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:53.925781 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.931535 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:53.937114 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:53.951115 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:11:53.960322 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:53.964210 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:54.131127 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:11:54.188212 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.205191 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.206475 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:54.208128 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.222951 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:54.307643 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:11:54.417546 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.431615 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:54.438792 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.447885 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:54.464267 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:54.464551 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:54.468909 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:54.603914 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:11:54.660961 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.675171 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.676266 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.676440 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:54.682394 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:54.725581 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:11:54.884255 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.890413 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:54.907747 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:54.910439 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:54.934780 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:54.943898 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:11:54.958709 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:55.116137 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:55.124286 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:55.151906 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:11:55.152897 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:55.168332 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:55.168481 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 14:11:55.172396 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:55.231169 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:55.278472 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 14:11:55.334004 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240501 14:11:55.428169 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:55.438127 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:55.440188 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:55.455008 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240501 14:11:55.458534 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:55.459833 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:55.573457 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:55.684585 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:11:55.701921 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:55.727293 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:55.749421 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:55.750824 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:55.829543 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:55.904547 19045 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:55.904753 19045 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:11:55.905810 19045 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:11:55.907101 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:55.911955 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:11:55.912254 30927 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:11:55.912463 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:11:55.912623 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:55.912875 19045 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:55.958829 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:55.965873 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:55.970057 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:55.984350 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:55.989391 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:56.062474 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:11:56.142841 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:11:56.159966 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:56.222702 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:56.231395 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:11:56.243238 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:56.248574 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:56.295812 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.302505 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:56.491854 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:56.505813 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:56.508283 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:56.515810 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:56.528358 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:56.734400 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:56.748250 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:56.753312 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:56.757455 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:56.768253 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:56.771674 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:56.799919 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:56.890548 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:11:57.003358 19045 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:57.003571 19045 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:11:57.004310 19045 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:11:57.005908 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:11:57.006413 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:57.006542 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:11:57.006852 19045 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:57.011529 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:57.022011 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:57.022099 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:57.034375 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:57.039292 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:57.059530 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:11:57.104725 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:11:57.140537 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:11:57.141764 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:11:57.269428 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:57.276647 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:11:57.279641 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:57.284364 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:57.289901 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.299808 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:57.319334 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:57.523418 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:57.527281 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:11:57.527980 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:11:57.530354 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:57.531899 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:57.537195 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:57.537992 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:11:57.714799 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 14:11:57.758837 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:11:57.763980 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:57.782795 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:57.784518 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:57.787425 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:57.796558 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:57.818307 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 14:11:57.825451 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:11:57.905253 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:58.042515 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.051569 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.058739 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:58.067798 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:58.068241 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:11:58.161669 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20240501 14:11:58.280592 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.283581 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:58.289691 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.312672 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.315531 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:58.437986 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:58.442704 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:58.494510 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:11:58.506239 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:11:58.540807 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.543674 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.544495 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:58.568861 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:11:58.595103 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.597537 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:11:58.803485 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.812310 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:58.813575 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:11:58.820097 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:58.826222 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:58.827638 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:58.853994 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:58.946550 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:11:59.080519 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.086442 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.094064 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:59.118387 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.119973 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:59.145409 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:11:59.201227 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:11:59.338519 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.351222 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.370853 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.380684 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:59.381048 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:59.409037 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:11:59.496291 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:11:59.597206 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 14:11:59.615303 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.621225 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:11:59.642597 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.677392 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.678128 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:59.689836 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:11:59.715867 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:11:59.811257 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:11:59.880474 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.882419 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:11:59.888186 19045 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:11:59.888404 19045 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:11:59.889339 19045 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:11:59.890188 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:11:59.891059 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:11:59.900434 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:11:59.900599 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:11:59.903396 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:11:59.903571 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:11:59.903852 19045 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:11:59.925331 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:11:59.989136 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:00.097626 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:00.112473 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240501 14:12:00.122949 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:00.123559 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.124946 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:00.131583 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:00.218160 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:12:00.229017 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 14:12:00.345031 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:12:00.346575 18230 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:00.346850 18230 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:00.348263 18230 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:12:00.348946 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:12:00.349221 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:12:00.350701 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:00.354996 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:00.355129 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:00.355594 18230 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:00.378310 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:00.396219 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:00.397883 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:00.412988 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:00.416555 18230 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:12:00.416767 18230 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:12:00.417558 18230 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:12:00.419520 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:00.419736 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:00.419857 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:00.420392 18230 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:00.439757 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:00.516428 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 14:12:00.548950 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:00.587627 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:12:00.650707 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:00.652801 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:00.653687 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:00.668440 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.673401 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:00.739928 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:12:00.913954 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:00.926143 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:00.929600 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:00.948588 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:00.953131 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:00.991508 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:01.037253 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:12:01.059388 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:12:01.207602 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:01.208807 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:01.218299 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:01.228901 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.243110 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:01.356915 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:01.457791 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:01.467177 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:01.470820 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:01.511305 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.511632 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:01.519240 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:12:01.720906 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:01.723013 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:01.744575 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:01.745488 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:01.746574 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:01.763728 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:01.925268 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:12:01.946297 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:12:01.979689 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:01.984125 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:01.985335 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:01.986829 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.015694 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:02.040114 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:12:02.049902 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:12:02.207614 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:12:02.221874 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:02.237991 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:02.240855 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:02.242543 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:02.269527 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:02.274075 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:12:02.303225 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:12:02.355589 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:02.392426 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 14:12:02.488296 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 14:12:02.520426 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240501 14:12:02.537098 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:02.539170 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:02.541018 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:02.553001 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:02.555598 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:02.778959 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:02.780745 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:02.784178 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:02.791081 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:02.791296 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:02.812265 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:12:02.877257 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 14:12:02.908766 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:03.003283 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:12:03.019191 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.029503 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.051544 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:03.055877 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:12:03.056023 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:03.059697 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:03.242264 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.252375 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.265709 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.273276 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:03.287089 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:03.287551 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:03.296476 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:03.475694 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:03.478138 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:03.515869 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.535332 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.539636 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:03.552403 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.564857 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:03.664108 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:12:03.773751 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.776930 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:03.779650 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:03.785526 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:03.787034 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:03.900471 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:03.901625 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:04.014457 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:04.018127 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:04.018760 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:04.046423 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.054157 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:04.054554 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:04.202818 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:12:04.299871 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:04.318518 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:04.386986 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:04.412289 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:04.444432 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:04.479641 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:04.500514 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 14:12:04.500741 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:04.562750 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:04.631292 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:04.651455 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:04.660255 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:04.676899 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:04.677439 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:04.683796 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:12:04.777384 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:04.900118 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:04.930176 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:04.937283 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:04.949959 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:04.951229 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:04.965350 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:04.979583 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:05.000717 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 14:12:05.140969 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240501 14:12:05.188678 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.196841 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.207413 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:05.216820 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:05.236713 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:05.340049 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:12:05.364629 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:05.430694 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.433282 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.441808 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:05.449961 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:05.470693 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:05.507112 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 14:12:05.555651 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:12:05.564427 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:12:05.678344 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.681497 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.682096 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:05.715451 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:05.738968 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:05.744125 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:05.866688 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:12:05.921677 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.942494 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:05.955932 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:05.958356 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:05.961699 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:06.024812 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:06.116973 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:12:06.211581 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:06.214936 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:06.220446 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:06.232116 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.242816 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:06.248080 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:06.462793 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:06.463984 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:06.482538 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:06.484175 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.491220 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:06.681452 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:06.681675 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:06.684779 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:06.710143 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.719707 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:06.750471 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:06.816402 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:06.836838 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:06.867636 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:12:06.910823 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:06.920415 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:06.927172 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:06.932703 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:06.937594 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:07.078734 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:12:07.112550 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:12:07.120568 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:12:07.142549 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.148298 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.169642 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:07.173625 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.184063 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:07.252562 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:07.373728 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:12:07.394996 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.408514 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.412004 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:07.428761 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.435495 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:07.458004 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 14:12:07.468247 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:07.502841 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:07.514649 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
I20240501 14:12:07.628293 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.645576 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.656363 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.656951 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:07.668484 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:07.693706 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:12:07.729022 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 14:12:07.740526 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:07.900225 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.908147 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:07.929340 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:07.931906 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:07.936436 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:08.152794 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:08.153784 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:08.160857 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.178416 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.187559 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:08.192039 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:08.221720 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:08.319595 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:08.341537 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:12:08.410725 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.412389 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.413002 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:08.425935 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.430119 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:08.476938 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:08.480670 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:08.585894 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:08.605904 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:08.654453 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.667884 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.673238 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:08.674922 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.679744 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:08.876062 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.886684 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:08.895567 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:08.901077 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:08.914772 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:08.980635 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:08.990967 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:09.052340 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:09.117566 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.119225 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.137297 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:09.142544 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:09.156847 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:09.263039 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:12:09.348078 20890 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:12:09.348297 20890 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:12:09.349228 20890 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:12:09.356290 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:09.366770 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:09.367048 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
W20240501 14:12:09.367347 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:12:09.367437 20890 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:09.392691 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.400470 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.405750 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:09.416029 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:09.428143 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:09.539723 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:09.569025 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:12:09.622464 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:09.652796 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:09.661010 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.665649 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.669708 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:09.683274 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:09.689410 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:09.769793 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:12:09.796968 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:09.878499 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240501 14:12:09.934944 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.955446 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:09.956348 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:09.984329 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:09.985534 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 14:12:09.987625 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:10.145089 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 14:12:10.186986 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:10.200822 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:10.212265 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:10.213750 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:10.222882 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.227353 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:10.301733 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240501 14:12:10.481889 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:10.489782 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:10.493515 20692 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:10.493703 20692 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:12:10.494472 20692 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:12:10.495445 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:12:10.496165 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:12:10.504299 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:10.504460 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:12:10.505023 20692 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:10.511626 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.519460 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:10.545830 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:10.576448 20692 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:10.576712 20692 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:10.577884 20692 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:12:10.580812 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:10.598955 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:12:10.599224 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:12:10.600428 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:10.600821 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:10.601490 20692 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:10.742799 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:10.743875 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:10.747821 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:10.766059 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:10.769744 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:10.786307 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 14:12:10.794014 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:10.812135 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:11.005228 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:11.006920 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.010516 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.011766 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:11.029896 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:11.044327 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:11.051192 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:11.252931 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.268512 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.269873 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:11.276227 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:11.289155 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:11.314900 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:11.334606 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:12:11.538394 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.542309 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:11.542719 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.548902 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:11.572321 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:11.643847 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:12:11.759424 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:12:11.772210 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:11.796252 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.799999 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:11.801791 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:11.813833 21328 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:12:11.814038 21328 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:12:11.815436 21328 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:12:11.819932 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:11.820259 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:12:11.820370 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:12:11.820786 21328 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:11.834081 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:11.837333 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:11.952622 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:12.041271 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:12.056792 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:12.061013 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240501 14:12:12.070091 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:12.072630 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.089740 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:12.092018 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:12.132889 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:12:12.158169 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:12.214308 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:12.288254 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.291095 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:12.291294 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:12.303131 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:12.328296 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:12.344908 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:12.393564 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:12.474370 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:12:12.526852 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 14:12:12.534813 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:12.555326 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:12.583202 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:12.586388 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.597890 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:12.624528 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 14:12:12.662664 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240501 14:12:12.815300 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:12.824021 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:12.831110 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:12.842253 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:12.863680 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:13.008870 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:13.090425 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.099936 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.101186 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:13.128100 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:13.133946 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.382525 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.385177 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.401068 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:13.408710 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:13.428014 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:13.445508 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:13.506938 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:13.514199 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:12:13.604569 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:13.605844 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:13.612416 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:13.643421 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.646108 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.662937 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:13.663280 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:13.667838 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:13.805429 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:12:13.904995 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:13.906729 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.907179 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:13.914258 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:13.929405 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:14.015535 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 14:12:14.071516 21843 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:12:14.071717 21843 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:12:14.072408 21843 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:12:14.075119 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:14.076355 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:12:14.076486 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:12:14.076769 21843 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:14.088089 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:14.103380 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:14.143790 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:12:14.156666 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:14.157469 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:14.161327 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:14.173928 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:14.186434 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:14.260169 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:14.373756 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:12:14.390027 21866 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:14.390285 21866 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:12:14.391091 21866 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:12:14.394440 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:14.405200 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:14.405413 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:14.405850 21866 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:14.446065 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:14.448318 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:14.454141 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:14.499572 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:14.502359 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.511978 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:14.616190 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:14.636212 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 14:12:14.672482 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:14.674568 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:12:14.726193 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:14.734542 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:14.735545 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:14.754104 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:14.758432 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:14.825477 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:12:14.854012 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240501 14:12:14.986426 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:15.005707 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:15.014290 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:15.029923 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.039636 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:15.074775 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:12:15.107671 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:15.271751 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:15.279677 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:15.291395 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:15.292706 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240501 14:12:15.301893 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:15.320667 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:15.378901 28940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.132:42827, user_credentials={real_user=slave}} blocked reactor thread for 94352.7us
W20240501 14:12:15.499069 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:12:15.537556 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:15.538890 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:15.548166 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:15.560448 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:15.568814 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.568898 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:15.776238 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:15.776846 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:15.798676 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:15.804729 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:15.806422 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:15.825019 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:12:15.956449 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:15.963718 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:16.010321 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:16.012949 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:16.022500 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:16.047478 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:12:16.063899 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.069679 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:16.267182 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:16.267912 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:16.287109 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:16.293960 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:16.302803 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:16.305069 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:12:16.441012 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:12:16.547304 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:16.566102 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:16.571223 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.571666 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:16.577701 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:16.596771 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:16.799583 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:16.809509 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:16.814437 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:16.820538 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:16.821036 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:16.863772 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:12:16.864673 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:12:16.883226 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240501 14:12:16.904845 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 14:12:17.080412 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:17.087715 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:17.095013 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:17.102808 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:17.114699 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.116983 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:17.196663 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:17.235682 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:12:17.276167 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:12:17.277333 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240501 14:12:17.312743 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:17.319614 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:17.335096 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:17.349604 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.391880 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:17.416013 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:12:17.590550 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:17.600008 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:17.611383 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.618423 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:17.622049 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:17.660288 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:17.696624 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:17.745352 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 14:12:17.794121 22274 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:12:17.794306 22274 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:12:17.795199 22274 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:12:17.797108 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:17.797291 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:17.797379 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:17.797660 22274 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:17.822100 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:17.831111 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:17.834363 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:17.843856 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:17.856770 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:17.958743 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240501 14:12:18.041327 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:18.049793 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:18.075592 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.079437 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:18.123481 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:18.210192 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:18.237553 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:18.295491 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:18.303930 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:18.304338 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:18.327201 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:12:18.331390 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:18.335800 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:18.417835 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:18.473274 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:12:18.580263 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:18.583448 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:18.602916 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.626610 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:18.650175 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:18.722077 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:18.849696 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:18.851929 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:12:18.877095 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:18.907869 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:18.912679 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:12:18.929354 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:18.931218 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:19.078146 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:12:19.127517 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:12:19.129287 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.138046 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:19.150156 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.155581 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.155968 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:19.192648 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:19.309173 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240501 14:12:19.367203 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:19.385751 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.395174 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.417207 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:19.435856 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:19.470371 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:19.508364 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:19.509320 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:19.596215 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:12:19.644405 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.647322 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.648923 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:19.675093 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:19.698149 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:19.735739 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:19.780956 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:19.788260 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:19.811316 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240501 14:12:19.915072 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.915793 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:19.921715 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:19.941450 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:19.942255 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:20.081195 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:12:20.119876 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:12:20.154747 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:20.167963 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:20.174131 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:20.183461 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:20.191462 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:20.193749 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:20.326001 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:12:20.442176 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:20.453805 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:20.464072 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.470155 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:20.477077 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:20.680002 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:20.690207 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:20.696569 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:20.743639 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:20.747859 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:20.750986 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:20.755818 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240501 14:12:20.919173 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:12:20.959326 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:20.996338 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:12:21.022560 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.023655 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.041083 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.044215 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:21.050930 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:21.072750 21866 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:21.072954 21866 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:12:21.073662 21866 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:12:21.082893 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:21.083062 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:21.083220 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:21.083667 21866 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:21.134434 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:12:21.219650 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:21.242055 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.260021 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.261665 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:21.281987 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:21.285845 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:21.478403 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:21.507925 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.511576 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:21.514979 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.526831 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:21.531685 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:21.726356 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:12:21.755343 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.755815 22274 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:21.756004 22274 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:21.756991 22274 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:12:21.757416 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:12:21.757745 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:12:21.762316 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:21.763438 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:21.763556 30862 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:21.764887 22274 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:21.773432 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:21.786945 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:21.791719 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:21.794811 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:21.828960 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:21.829790 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240501 14:12:21.909750 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:21.934568 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:12:22.010186 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:22.020439 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:22.035696 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.036132 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:22.058080 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:22.185897 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240501 14:12:22.267390 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:12:22.282884 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:22.289397 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:22.295811 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:22.337462 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:22.340005 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:22.371271 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:22.395020 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:12:22.539017 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:22.540815 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:22.557729 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:22.565392 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:22.578320 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:12:22.585371 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.608405 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:22.701478 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 14:12:22.761354 21866 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:22.761561 21866 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:12:22.762396 21866 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:12:22.763005 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:12:22.764365 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:22.764588 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:12:22.764748 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:12:22.765074 21866 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:22.799939 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:22.817976 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:22.821888 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:22.833437 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:22.854489 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:22.873250 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:23.055205 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:23.066452 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:23.075815 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:23.090418 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:23.107990 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.113498 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:23.139640 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 14:12:23.233762 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240501 14:12:23.331693 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:23.343940 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:23.358244 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:23.363169 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:23.382316 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:23.416128 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:23.504099 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 14:12:23.582283 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:23.584371 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:12:23.585429 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:23.597493 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:23.598477 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.601670 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:23.619925 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:23.636986 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:23.806686 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:23.814323 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:23.823134 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:23.839524 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:23.847168 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:23.942535 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:24.025141 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:12:24.034265 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:24.098448 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:24.104130 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:24.104540 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:24.116745 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.131843 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:24.172474 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 14:12:24.180017 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 48069.5us
W20240501 14:12:24.195045 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fb04ec77da194832aa63d69aba307de3: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:24.241531 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:24.241943 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:12:24.323446 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:12:24.332439 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240501 14:12:24.401743 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:24.404847 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:24.408917 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:24.429239 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:24.430486 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:24.503898 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:12:24.573601 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240501 14:12:24.630818 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:24.646746 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20240501 14:12:24.647420 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:24.648840 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:24.650849 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:24.653823 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.689105 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:24.766501 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:24.868736 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:24.869315 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:24.891549 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:24.892731 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:24.930712 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:24.985963 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 103173us
W20240501 14:12:24.988149 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:25.077188 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:12:25.121788 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:12:25.158326 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:25.198714 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:25.199206 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.201438 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.208427 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.216251 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:25.361322 28589 catalog_manager.cc:4573] TS 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 135da52054584a27959efdd9126cfcef: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:25.416041 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:25.421547 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.430783 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.443877 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:25.446949 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.462083 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:25.712085 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 14:12:25.718537 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:25.734191 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.737787 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.741544 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:25.747638 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:25.779891 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:25.847783 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:12:25.855418 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:25.971004 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:25.977491 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.979569 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:25.994222 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.008059 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:26.181895 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:12:26.201804 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:26.202102 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:26.223675 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:26.235002 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:26.235776 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:26.250432 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.263020 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:26.459555 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:26.466188 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:26.487630 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:26.494290 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.508184 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:26.510480 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:26.596920 22274 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:26.597141 22274 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:26.597975 22274 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:12:26.599897 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:12:26.600134 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:12:26.601593 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:26.618207 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:26.618489 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:26.618906 22274 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:26.655313 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:12:26.742733 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:26.760222 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:26.771055 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:26.773774 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:26.782945 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:26.800601 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:26.983047 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240501 14:12:26.993479 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:26.998080 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:27.001647 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:27.022576 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.024574 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:27.061548 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:12:27.103478 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:12:27.133747 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 14:12:27.190302 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:27.234205 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:27.238729 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:27.239046 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:27.261600 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:27.271237 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.295367 21866 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:27.295574 21866 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:12:27.296334 21866 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:12:27.298893 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:27.300438 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:27.300541 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:27.301064 21866 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:27.343266 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:27.488989 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:27.491443 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:12:27.524245 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:27.528743 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:27.530131 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:27.541113 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:27.541368 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:27.557173 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:27.641007 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:12:27.775314 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:27.779300 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:27.780508 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:27.784018 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:27.798717 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:27.830024 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:27.886880 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:28.022390 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:28.030743 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:28.050748 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.050837 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:28.055817 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:28.176012 23984 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:12:28.176208 23984 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:12:28.176944 23984 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:12:28.178458 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:28.179000 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:12:28.179100 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:12:28.179337 23984 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:28.272907 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:28.276608 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:28.278194 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:28.283463 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240501 14:12:28.290032 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.312234 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:28.365046 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:28.366199 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:28.516048 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:28.517596 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:28.524202 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:28.528133 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:28.543116 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.552773 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:28.572566 22274 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:12:28.572777 22274 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:12:28.573521 22274 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:12:28.598665 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:12:28.610414 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:28.610617 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:28.610716 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:28.610980 22274 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:28.732841 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:28.785168 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:28.786299 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:28.788621 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:28.806478 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:28.808154 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:28.844009 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:12:28.963188 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:29.023599 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:29.030022 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:29.045089 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:12:29.045167 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:29.085183 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.111500 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:29.256963 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:29.279356 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:29.307476 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:29.313805 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:29.319368 22274 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:29.319568 22274 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:29.320451 22274 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:12:29.321872 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:12:29.322124 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:12:29.327486 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:29.339720 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:12:29.350039 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:29.354287 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:29.354343 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:29.355177 30862 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:29.355562 22274 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:29.387092 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:29.427533 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240501 14:12:29.533344 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:12:29.565825 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:29.594311 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:29.598094 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:29.603713 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:29.615038 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:29.643294 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240501 14:12:29.652563 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:29.684044 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:29.741344 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:29.742871 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:12:29.814404 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:29.843801 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:29.858119 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:29.869323 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:29.890287 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:29.892306 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:30.017966 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:30.089337 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:30.094302 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:30.096748 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:30.109508 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:30.123093 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:30.126936 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:12:30.139761 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.330889 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:30.343068 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:30.347311 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:30.379611 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:30.392155 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:30.456885 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:12:30.567905 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:30.602983 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:30.627048 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:30.632050 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:30.638139 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:30.650884 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:30.664355 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 14:12:30.680410 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:30.872516 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:30.876507 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:30.888126 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:30.899348 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:30.919718 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:31.036026 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:31.075860 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:12:31.138804 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:31.182389 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:31.183454 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.188145 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.194875 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.198894 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:31.318590 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:12:31.421727 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.423702 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.437386 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:31.442790 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:31.447774 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:31.462323 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:31.672960 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.679858 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.701879 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.709765 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:31.725303 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:31.768683 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:31.820499 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:12:31.930975 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.934713 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:31.942454 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:31.956125 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:31.958734 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:31.960184 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:32.043532 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:12:32.054442 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 14:12:32.121487 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:12:32.161152 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:12:32.169543 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240501 14:12:32.178573 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:32.179937 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.192406 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.197646 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.198968 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:32.246977 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:12:32.252137 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:32.287827 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e0bf08cbb6d347d783b1fa7762c802df: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:32.349797 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:32.388067 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 14:12:32.431165 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:32.441241 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.458295 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.474526 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:32.495762 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:12:32.521742 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:12:32.525281 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:32.683562 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:32.685035 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.703907 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.711921 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:32.730516 21866 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:32.730741 21866 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:12:32.731503 21866 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:12:32.733809 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:32.738276 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:32.757416 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:32.757695 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:32.758008 21866 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:32.828716 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:32.957751 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.966737 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:32.974435 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:32.976532 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:32.997236 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:33.031131 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:12:33.086975 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:12:33.198066 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:33.204859 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:33.216435 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.218611 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:33.225390 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:33.243152 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:33.325500 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240501 14:12:33.417472 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:12:33.459705 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:33.470995 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:33.479666 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:33.513271 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.525552 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:33.565332 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:12:33.722553 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:33.727779 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:33.729533 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:33.742024 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:33.763767 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:33.776620 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:33.986366 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:33.994377 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240501 14:12:33.996920 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:34.025827 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:34.039963 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:34.040928 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:34.047662 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:34.072227 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:34.195411 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:12:34.229863 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:34.231032 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:34.235150 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:34.255021 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:34.257045 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:34.277760 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:34.319101 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:34.320005 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:34.521587 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:12:34.543228 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:34.547839 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:34.551230 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:34.557699 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:34.566671 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:34.742327 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:34.743899 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:12:34.772555 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 14:12:34.772904 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:34.786821 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:34.789947 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240501 14:12:34.791247 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:34.799341 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:34.803136 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:34.805626 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240501 14:12:34.807888 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:34.922163 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:12:34.946859 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:34.996338 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:12:35.047917 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.060256 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.063443 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:35.085510 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.102891 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:35.294361 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:35.334509 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.339717 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:35.345974 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.394011 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:35.394495 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:35.400736 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 14:12:35.609405 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.633198 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:35.643311 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:12:35.643985 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.662354 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.672305 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:35.746591 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:35.768631 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:35.825685 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240501 14:12:35.891047 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.893712 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:35.905226 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:35.906657 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:35.914063 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:36.122447 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.130062 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.135187 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:36.146005 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:36.157307 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:36.207870 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:12:36.251525 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:36.410831 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.444118 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:36.445416 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:36.452859 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:36.460237 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.463244 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:36.540400 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:12:36.678745 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.698225 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:36.702569 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.703699 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.704936 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:36.720036 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:12:36.792346 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:36.926540 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:36.945680 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.946393 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:36.959918 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:36.977113 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:37.069768 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240501 14:12:37.170318 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:12:37.174773 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:12:37.208530 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:37.215620 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:37.222282 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:37.231726 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:37.237095 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:37.247496 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:37.248379 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240501 14:12:37.265223 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.486925 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:37.501303 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:37.508814 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:37.515175 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.520087 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:37.521361 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:12:37.571153 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:12:37.633692 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:37.670367 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:12:37.738876 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:12:37.763398 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:37.768755 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:37.775305 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:37.778507 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:37.791497 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:37.846769 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:12:37.992107 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240501 14:12:37.993477 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:37.995798 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:37.996388 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:38.004447 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:38.021248 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:38.038106 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:38.180820 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:12:38.191948 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:38.260843 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:38.271656 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:38.273954 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:38.290776 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:38.340837 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:38.424212 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:38.533056 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:38.534101 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:38.542338 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:38.563417 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:38.564428 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:38.565286 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.767985 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:38.779290 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:38.791756 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:38.805318 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:38.814831 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:38.854223 25110 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:12:38.854418 25110 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:12:38.855170 25110 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:12:38.876852 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:38.877089 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:38.877214 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:38.877477 25110 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:39.014106 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:39.015529 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:39.024883 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:39.029697 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:39.031510 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:39.039332 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:39.042399 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:39.053851 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:39.069782 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:12:39.145236 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:39.237454 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:12:39.262441 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:39.266919 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:39.281034 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:39.286250 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:39.302976 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:39.425376 25110 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:39.425643 25110 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:39.426919 25110 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:12:39.427783 29136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:12:39.428035 29136 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:12:39.428817 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:39.429550 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:39.429704 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:39.429982 25110 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:39.434296 25153 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:39.434465 25153 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:12:39.435319 25153 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:12:39.435990 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:12:39.436853 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:12:39.438480 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:39.438609 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:12:39.439175 25153 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:39.504734 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:12:39.538738 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:39.544586 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:39.553001 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:39.553193 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:39.565824 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:39.580106 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:39.627460 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:12:39.631175 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240501 14:12:39.673431 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 14:12:39.780896 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:39.783108 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:39.788916 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:39.793097 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240501 14:12:39.801781 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:39.809976 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:39.924731 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:39.925676 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:12:39.928416 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:12:40.027951 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:12:40.054174 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:40.054881 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.057972 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.059185 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:40.073546 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:40.144129 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:12:40.190658 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:40.246999 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:40.319830 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.324780 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:40.349195 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.350708 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:40.369681 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:40.411465 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240501 14:12:40.546550 25359 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:12:40.546811 25359 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:12:40.547641 25359 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:12:40.550760 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:40.552472 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:12:40.552610 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:12:40.552860 25359 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:40.589723 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:40.600523 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.601480 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.615620 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:40.618974 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:40.726505 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:40.790022 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:12:40.808089 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:12:40.825889 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.834964 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:40.840835 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:40.844488 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:40.862514 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:40.894899 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:41.096714 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.107156 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.110913 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:41.113610 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.126293 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:41.159793 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:41.326884 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.327067 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.334678 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:41.388118 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.395582 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:41.421401 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:41.560171 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:41.589569 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:12:41.628386 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.653821 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:41.655352 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.667318 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.668560 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:41.839440 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:12:41.929114 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.937474 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:41.937932 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:41.941201 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:41.953184 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:42.000805 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:12:42.027119 25153 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:42.027710 25153 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:42.029237 25153 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:12:42.032048 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:12:42.032410 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:12:42.037616 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:42.038517 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:42.038715 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:42.039099 25153 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:42.045650 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:42.047879 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:12:42.172197 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:42.194129 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:42.194846 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:42.197970 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:42.199119 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:42.278801 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:12:42.352972 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240501 14:12:42.411607 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:42.412254 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240501 14:12:42.425042 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:42.429400 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:42.434285 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:42.450439 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:42.450963 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:42.456552 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:12:42.489706 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:12:42.577759 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240501 14:12:42.658715 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:42.676175 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:42.704747 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.715442 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:42.737236 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:42.756201 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:12:42.824676 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:42.887277 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:42.913934 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:42.927295 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:42.932868 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:42.954655 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:42.972707 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:43.013876 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240501 14:12:43.037550 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:12:43.161880 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:43.172791 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:43.179507 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:43.180505 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.198069 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:43.287477 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:43.419112 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:43.433635 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:12:43.433943 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:43.437603 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:43.441778 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:43.452360 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:43.563608 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:12:43.631871 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:12:43.718669 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:43.730901 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:43.730998 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:43.737039 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:43.738921 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:43.883265 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:43.998796 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.021281 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:44.026229 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:44.035987 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.040242 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:44.239078 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:12:44.248327 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:44.256632 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.260089 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.272330 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:44.274076 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:44.283617 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:44.327239 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:44.357518 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:44.366896 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:44.472786 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240501 14:12:44.502068 25153 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:44.502270 25153 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:12:44.503094 25153 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:12:44.507337 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:44.507670 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:44.507771 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:44.508670 25153 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:44.523764 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.539041 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:44.542682 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.554854 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:44.567746 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:44.662063 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:44.663398 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 14:12:44.777065 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.794240 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:44.795889 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:44.802487 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:44.827294 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:44.870000 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:12:44.883930 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240501 14:12:44.893846 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:44.927645 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:12:44.957093 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:12:44.972414 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240501 14:12:45.031822 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:45.032708 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:45.062615 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:45.062929 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:45.085496 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:45.192459 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:12:45.297255 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:45.306939 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:45.316216 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240501 14:12:45.318318 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:45.319857 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:45.329355 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.354779 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:45.464334 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:12:45.514430 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:45.552248 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240501 14:12:45.552942 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:45.598682 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:45.640564 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:45.648452 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:45.692739 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.709079 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:45.856343 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:45.856988 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:45.878327 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:45.901484 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:45.912585 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:45.995380 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:46.097007 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:46.123935 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.132815 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.145193 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:46.146955 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:46.167436 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:46.278483 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:12:46.397500 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.404898 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.405552 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:46.417382 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.419261 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:46.485669 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:46.625228 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:46.631708 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.632463 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.656987 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.657761 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:46.660804 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:46.864187 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:12:46.869251 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:46.874771 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.876058 26734 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:46.876260 26734 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:46.877180 26734 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
I20240501 14:12:46.878289 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "a0e901103604414c829b40edf9f24c28" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:12:46.878563 29138 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e901103604414c829b40edf9f24c28 in term 2.
W20240501 14:12:46.881469 30862 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:46.882972 30863 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:46.883098 30863 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:46.883916 26734 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:46.884660 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 14:12:46.898665 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:46.920032 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:46.934242 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:46.934911 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:47.003870 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:47.122771 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.137120 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:47.139438 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.153892 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.178874 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:47.362138 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240501 14:12:47.409771 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240501 14:12:47.410249 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.411916 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.428438 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:47.433909 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:47.442317 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:47.453094 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:47.479984 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:47.577817 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:12:47.578766 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:47.638931 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:12:47.657308 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:47.668790 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.671406 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.686537 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:47.690243 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:47.694248 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:47.923090 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.930156 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:47.935294 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:47.948627 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:47.951684 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:12:47.952080 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:48.025251 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:48.040606 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:48.114979 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:48.137699 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240501 14:12:48.181093 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.185071 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.186053 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:48.190944 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:48.210455 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.381827 27179 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:48.382058 27179 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:48.383086 27179 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:12:48.384568 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:48.384517 30925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:12:48.384809 30925 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:12:48.384967 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:48.385116 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:48.385356 27179 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:48.411235 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:12:48.422811 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:48.423280 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.447414 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.456148 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.456174 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:48.475482 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:48.672793 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:48.678503 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.694949 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.697834 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.707659 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:48.884472 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:48.934376 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:48.941691 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.945850 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:48.951078 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:48.958165 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:48.980662 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:49.076936 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 14:12:49.212807 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:49.213475 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:49.225289 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:49.233246 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.243054 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:49.282151 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:12:49.445984 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:49.469709 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:49.473423 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:49.475929 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:49.477993 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:49.480903 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:49.542682 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:49.548270 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:49.599550 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:12:49.620436 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240501 14:12:49.663461 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:12:49.695667 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:49.700651 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:49.706527 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:49.708424 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:49.716006 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:49.938956 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:49.941025 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:49.943297 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240501 14:12:49.948113 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:49.955994 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:49.960227 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:49.972980 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:49.982249 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:49.998129 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:50.036074 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:50.121323 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:50.161480 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 14:12:50.197614 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:50.202232 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:12:50.202401 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:50.203904 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:50.214597 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:50.243328 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.439890 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:50.447803 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:50.464669 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.464907 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:50.465492 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:50.487488 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:50.499475 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:12:50.506012 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240501 14:12:50.570463 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:50.707487 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:50.754295 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:50.754298 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:50.760109 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:50.828079 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:50.894220 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:12:50.901067 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:50.990471 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:50.997956 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:50.998617 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:51.000133 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.023196 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:51.043884 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.230579 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.243151 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.259929 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.260198 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:51.264919 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:51.283653 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:12:51.323752 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:51.409307 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:51.506482 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.507544 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.523439 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:51.526291 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.528470 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:51.742313 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.761329 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:51.763908 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.764843 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:51.773880 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:51.865324 27738 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:12:51.865514 27738 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:12:51.866250 27738 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:12:51.868386 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:51.871670 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:12:51.871806 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:12:51.872071 27738 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:51.984992 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.986342 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:51.991657 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:51.998286 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:12:52.011059 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:52.020645 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:52.023519 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:52.071082 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:12:52.181221 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:52.237706 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:52.245167 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:52.260617 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:52.277388 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:52.287973 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:52.295625 27847 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:52.295831 27847 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:12:52.296749 27847 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
I20240501 14:12:52.298492 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:12:52.298774 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:12:52.299212 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:12:52.299368 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:52.299456 29072 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:52.299911 27847 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:52.304646 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:52.371299 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:52.404786 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240501 14:12:52.464708 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240501 14:12:52.531037 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:52.531875 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:52.538368 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:52.551822 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:52.555925 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:52.615165 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:12:52.633020 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:52.655448 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:12:52.718705 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:12:52.769657 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:52.770799 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:52.779227 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:52.793916 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:52.815912 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:52.874738 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:12:52.944132 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:53.019093 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.022818 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:53.026782 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.035393 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:53.067513 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:12:53.068809 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:53.089295 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:53.200783 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20240501 14:12:53.257287 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.264267 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:53.271203 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:53.274489 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.277844 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:53.387816 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:53.518328 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.530565 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.530896 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:53.542807 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.543044 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:53.663786 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:53.698463 28016 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:53.698714 28016 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:12:53.699542 28016 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:12:53.701234 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:53.701604 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:12:53.701714 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:53.701984 28016 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:53.769490 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.778241 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:53.785233 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:53.800835 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:53.802990 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:53.903394 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:54.018091 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:54.036373 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:54.065755 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:54.067175 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.071375 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:54.085450 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:54.176195 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:12:54.212443 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:12:54.309888 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:54.325700 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:54.334996 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:54.352659 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:54.371045 26734 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:12:54.371263 26734 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:12:54.372179 26734 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
I20240501 14:12:54.375769 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:54.401912 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:12:54.402225 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:54.402344 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:12:54.402698 26734 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:54.412653 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:54.452406 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:54.574257 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:54.618803 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:54.628736 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:54.642869 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:54.653518 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:54.671010 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:54.677629 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 14:12:54.680632 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:54.707767 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:12:54.854666 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:12:54.862480 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20240501 14:12:54.873746 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:54.887787 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:54.890178 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:54.914891 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:54.947669 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:54.991065 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:12:55.028407 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 14:12:55.095629 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:55.097582 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.106396 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:55.111230 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.129876 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:55.142704 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:55.144407 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240501 14:12:55.145962 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:12:55.221181 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:55.336771 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.347476 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.367580 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:55.422084 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.433219 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:55.459846 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:12:55.462613 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240501 14:12:55.571180 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:55.605335 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.605840 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.632815 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.634653 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:55.635318 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:55.705348 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:12:55.756179 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240501 14:12:55.757113 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:12:55.866700 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.868453 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:55.883145 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:55.884819 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:55.888839 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:56.054067 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:56.128788 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.133710 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:56.136210 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.152797 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.159760 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:56.299957 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:12:56.321203 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:56.382409 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.384682 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.416728 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:56.422652 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:56.443899 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:56.562188 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:56.673730 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.680351 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.701678 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:56.731812 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:56.737445 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:56.899066 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:12:56.932055 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.935456 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:56.941365 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:56.943331 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:56.944926 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:12:56.968333 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:57.145390 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:57.170365 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.180462 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.193229 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.199003 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:57.202215 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:57.264127 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:12:57.274477 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:12:57.284927 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:57.426488 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.432370 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.443727 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:57.447955 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.450112 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:57.466961 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:12:57.508416 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240501 14:12:57.513394 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240501 14:12:57.590261 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:12:57.599126 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:12:57.621541 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:57.689548 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.709393 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:57.711411 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.714183 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:57.717489 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:57.750998 28811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:12:57.752840 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:12:57.949889 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.952123 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:57.965443 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:57.973012 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:57.977360 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:58.057178 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240501 14:12:58.151727 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:58.185107 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.188730 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.196919 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:58.206262 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:58.207180 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:12:58.214169 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:58.419807 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:12:58.420913 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:58.427357 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.442533 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:58.448922 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.458076 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.681480 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.694079 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.696831 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:12:58.703243 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:12:58.712282 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.716141 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:58.820470 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:12:58.933319 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:58.934494 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.939886 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:58.951818 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:58.969077 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:59.124132 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:12:59.168887 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.172580 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:59.179806 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.194531 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:59.199980 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:59.217144 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:59.379037 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:12:59.416138 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.416774 29217 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:12:59.416960 29217 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:12:59.417842 29217 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:12:59.419483 28875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:12:59.419956 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:59.420356 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:12:59.420492 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:12:59.421031 29217 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:12:59.421337 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:59.421730 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.438127 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:12:59.444483 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:12:59.633630 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:12:59.663743 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 14:12:59.695432 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:12:59.696329 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.697353 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.715317 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.717339 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:12:59.750191 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:12:59.769333 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:12:59.776100 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:12:59.793900 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:12:59.906401 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240501 14:12:59.921031 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 14:12:59.924296 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:12:59.934674 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.939683 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:12:59.966115 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:12:59.981029 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:00.000563 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:00.048055 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:13:00.160523 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:00.164204 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240501 14:13:00.179430 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:13:00.189566 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:00.196295 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:00.202297 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:00.204593 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:00.215256 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.222851 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:00.257126 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:00.367434 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240501 14:13:00.426566 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:00.437536 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:00.439420 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:00.467382 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:00.468525 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:00.639122 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:00.687451 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:00.700302 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:00.703102 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:00.710296 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:00.730680 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:00.745484 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:13:00.919845 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:00.926188 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:13:00.930466 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:00.945660 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:00.982976 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:00.985903 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:01.027702 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:01.171895 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:13:01.246228 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:01.246289 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:01.254374 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:01.259332 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:01.261900 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:01.346493 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:13:01.417153 29952 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:13:01.417389 29952 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:13:01.418109 29952 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:13:01.419871 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:13:01.420013 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:13:01.420133 28810 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:13:01.420375 29952 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:01.520084 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:01.526867 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:01.538117 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:01.550078 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:01.565464 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:01.644332 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:01.733141 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:13:01.752067 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:13:01.792881 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:01.796466 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:01.797513 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:01.815524 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:01.819453 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:01.920487 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 14:13:02.037001 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:02.071074 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:02.100430 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:02.105677 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:13:02.121551 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:02.130704 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:02.176887 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:13:02.240195 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:02.296819 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:02.356484 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:02.383890 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:13:02.398955 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:02.403802 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:02.408212 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:02.424641 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:02.516719 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240501 14:13:02.592092 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:13:02.594041 30097 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:02.594287 30097 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:13:02.595345 30097 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:13:02.597117 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:13:02.600415 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:02.600584 29072 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:02.601370 30097 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:02.638602 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:02.640858 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:02.661525 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:02.683988 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:02.691179 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:02.695588 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:13:02.713438 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:02.765264 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:13:02.853374 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:02.919063 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
I20240501 14:13:02.935014 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:02.940724 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:02.946273 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:02.952054 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:02.952471 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:03.040185 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240501 14:13:03.148211 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:03.190856 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.195412 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.202298 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:03.203819 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:03.215055 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:03.223567 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:03.237048 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:03.448750 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.458945 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.459430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:03.479247 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:03.483180 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.687222 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.697252 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.703163 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:03.706377 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:03.710987 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:03.751197 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:03.917306 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:03.934725 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.949455 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:03.957430 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:03.962878 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:03.966725 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.193163 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:04.201750 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:04.204084 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:04.208132 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:04.215427 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:04.222977 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:04.249619 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:04.260376 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:13:04.289325 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:13:04.424454 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:04.512265 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:04.512553 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:04.520474 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.521901 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:04.522953 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:04.649757 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:13:04.651276 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240501 14:13:04.686692 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:04.687597 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:13:04.702913 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:04.759442 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:04.770680 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:04.775451 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:04.775552 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:04.777928 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:04.858783 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:13:04.953987 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:13:05.038661 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.052608 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.105289 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:05.106073 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:05.124022 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240501 14:13:05.130089 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:05.203048 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:13:05.281358 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:05.295540 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:05.310348 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.316632 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.324831 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:05.333134 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.356014 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:05.474601 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 14:13:05.475972 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:13:05.548214 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.565995 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:05.566788 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.572407 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:05.589232 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:05.683940 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:13:05.701455 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:13:05.782665 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.789806 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:05.803117 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:05.827052 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:05.834872 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:05.886307 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:05.904744 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
I20240501 14:13:06.054906 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.080215 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.087540 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:06.109860 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:06.121743 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.242916 30097 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:06.243144 30097 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:13:06.244277 30097 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:13:06.244983 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240501 14:13:06.246309 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:06.248873 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:13:06.249179 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
W20240501 14:13:06.251385 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:06.251634 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, a0e901103604414c829b40edf9f24c28; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:06.251973 30097 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:06.317389 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.321398 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:06.332892 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.361341 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.395691 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:06.470388 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:06.480121 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 14:13:06.599202 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.604732 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.614271 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:06.625845 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:06.626994 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:06.631722 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:13:06.841930 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:13:06.855253 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:06.869309 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.879820 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:06.890100 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:06.895871 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:06.914405 28679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 59184.9us
W20240501 14:13:06.924532 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:13:06.925168 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:06.968700 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:07.124461 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:07.137131 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:07.142181 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:07.147735 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:13:07.153476 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:07.155041 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240501 14:13:07.167833 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:07.255714 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:13:07.283341 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:07.370647 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:13:07.371520 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:07.384752 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:07.402468 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:07.416676 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:07.418727 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:13:07.419508 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:07.642385 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240501 14:13:07.655406 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:07.667992 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:07.685650 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:07.696352 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.704967 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:07.723007 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:07.883910 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 14:13:07.902541 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:07.905560 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 14:13:07.909057 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:07.943804 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:07.945852 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:07.969727 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:07.979939 28302 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:13:07.980137 28302 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:13:07.981083 28302 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:13:08.022987 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:13:08.023380 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:08.023510 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:08.023840 28302 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:08.023921 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:13:08.143739 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:08.170334 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.179173 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.182739 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:08.199851 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.200204 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:08.359342 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:08.389151 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:13:08.435156 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.448704 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:08.455057 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.466037 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:08.469506 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:08.544566 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:08.556670 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240501 14:13:08.708755 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.725124 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.726049 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:08.743341 30097 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:08.743521 30097 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:08.744227 30097 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:13:08.745622 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:13:08.746315 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
I20240501 14:13:08.751685 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:08.752629 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:08.764305 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:08.764482 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:13:08.765008 30097 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:08.962072 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.983811 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:08.984328 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:08.996987 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:13:08.997704 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:09.001345 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:09.051821 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:13:09.056257 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:09.111531 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:13:09.220280 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:09.229126 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:09.233850 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:09.243789 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:09.247377 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:09.279296 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:09.465260 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:09.510987 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:09.512116 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:09.525388 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.528930 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:09.535712 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:09.589550 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:09.669046 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:13:09.684429 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:09.715785 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240501 14:13:09.770236 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:09.778930 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:09.786233 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:09.801527 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:09.808915 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:09.848311 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:13:09.870975 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:09.969302 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:09.970245 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:13:10.031991 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.033334 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.033739 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.035264 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:10.040241 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:10.076022 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240501 14:13:10.270155 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:10.274909 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.275846 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.279281 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.291123 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:10.412770 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:13:10.437135 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:10.438254 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:10.485616 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:13:10.554045 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.556340 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.567217 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:10.574925 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:10.585274 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:10.671465 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 14:13:10.679803 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:13:10.755503 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:13:10.805460 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.806934 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:10.816824 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:10.828687 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:10.832264 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:11.046428 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:11.066987 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:11.074656 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:11.086808 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.097157 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:11.101537 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240501 14:13:11.117626 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:11.144737 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:13:11.247107 29072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 14:13:11.334043 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:11.342114 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:11.358305 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:11.371309 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:11.375012 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:11.548050 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:13:11.605321 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:11.616832 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:11.623818 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:11.632933 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:11.647279 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:11.665855 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:11.789698 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 14:13:11.863799 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:11.866466 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:11.867935 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:11.887876 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:11.891371 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:11.942891 30097 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:11.943113 30097 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:13:11.943847 30097 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:13:11.945870 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:11.953380 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:11.953603 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:11.953938 30097 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:12.117390 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.120606 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.121394 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:12.127400 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:13:12.139065 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:12.144019 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240501 14:13:12.170046 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:12.172504 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:12.187000 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:12.198797 30862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.134:44241, user_credentials={real_user=slave}} blocked reactor thread for 85007.2us
I20240501 14:13:12.243156 28302 raft_consensus.cc:482] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 826ccf8358b6453d88bcb132290b82a5)
I20240501 14:13:12.243352 28302 raft_consensus.cc:504] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } }
I20240501 14:13:12.244081 28302 leader_election.cc:288] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399), 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241)
W20240501 14:13:12.255554 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:13:12.255877 30862 leader_election.cc:334] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:12.255988 30862 leader_election.cc:302] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0e901103604414c829b40edf9f24c28; no voters: 826ccf8358b6453d88bcb132290b82a5, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:12.256311 28302 raft_consensus.cc:2726] T e8b9db597e654ed48b679abf58b8a2c8 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:12.272258 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:13:12.287073 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:13:12.299713 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:13:12.326196 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:13:12.424801 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.429327 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.433548 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:12.455025 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:12.474380 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:12.575233 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:12.664885 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240501 14:13:12.673902 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.679375 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:12.682793 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.700287 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:12.705420 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:12.747761 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:13:12.850903 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:12.889972 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 14:13:12.936384 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.939352 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:12.944217 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:12.946508 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:12.959009 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:12.965801 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:13.014760 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 14:13:13.193230 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:13.198859 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:13.208645 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:13.208643 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:13.228843 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:13.321220 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:13:13.373751 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:13.421187 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:13:13.449142 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:13.460886 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:13.475306 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:13.527015 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:13.528383 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:13.707742 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240501 14:13:13.737998 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:13.749935 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:13.752923 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:13.763669 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:13.769011 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:13.827777 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:14.025341 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.025799 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:14.047147 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.051837 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:14.064575 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.284799 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.291888 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.295135 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:14.297828 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:14.308487 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:14.346185 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:14.383145 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240501 14:13:14.421792 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:14.520188 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:13:14.523962 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.524202 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.525326 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:14.558743 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:14.561722 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:14.572136 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:13:14.761201 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.761175 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:14.776933 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:14.784155 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:14.795234 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:13:14.807807 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:14.820323 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:14.834848 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:13:14.835441 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:13:14.919768 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:13:14.975502 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:13:15.009936 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.011629 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:15.030681 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.050850 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:15.073901 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:15.129042 28941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:15.133399 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240501 14:13:15.291415 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:15.293588 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.302295 30097 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:15.302498 30097 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:13:15.304205 30097 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:13:15.313292 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:15.314684 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:15.314833 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:15.315507 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.316154 30926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdedd39e12a49659d989f37b3287d87" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "a0e901103604414c829b40edf9f24c28" is_pre_election: true
I20240501 14:13:15.316387 30926 raft_consensus.cc:2454] T 1fdedd39e12a49659d989f37b3287d87 P a0e901103604414c829b40edf9f24c28 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3972b170c54344a4ba61f176f837ba in term 2.
I20240501 14:13:15.317378 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:15.318939 30097 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:15.329594 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:15.343205 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:15.361873 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:13:15.371773 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:13:15.390045 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:13:15.444167 28679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 14:13:15.551367 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.556349 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:15.562726 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.563364 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:15.578374 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:15.829133 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.830677 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:15.850922 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:15.860960 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:15.868592 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:15.908093 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:13:15.942405 30862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 14:13:15.952263 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:15.968602 32332 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:15.968801 32332 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:15.969689 32332 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:13:15.970597 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7f2a8d8a84faf86c4de16b3cc6663" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd3c04ebb6c84b15a1b3adc748737094" is_pre_election: true
W20240501 14:13:15.971252 29071 leader_election.cc:341] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Illegal state: cannot vote while shut down
W20240501 14:13:15.971412 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:15.971562 29072 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:13:15.971835 32332 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:15.980417 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:13:16.074183 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:16.082010 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:16.096172 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.101388 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:16.135825 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:16.246646 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240501 14:13:16.305778 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:16.315052 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:16.334785 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:16.335690 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:16.343258 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:16.359172 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240501 14:13:16.400363 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:16.567518 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:16.572973 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:16.596457 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:16.609097 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.624186 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:16.693814 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240501 14:13:16.834486 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:16.835619 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:16.837482 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:16.840863 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:16.842315 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:16.874783 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:17.032109 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:13:17.049870 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:17.056835 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:13:17.059114 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:17.080783 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:13:17.085718 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:17.115370 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.127768 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:17.246069 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:13:17.305930 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:13:17.332383 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:17.335294 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:17.350847 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:17.372239 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:13:17.373037 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:17.373834 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:17.380378 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:17.441326 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:13:17.552691 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:13:17.622248 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:17.642601 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:17.643047 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:17.670840 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:17.695752 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:17.705372 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240501 14:13:17.759642 28941 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:13:17.771809 30862 consensus_peers.cc:485] T ec1b3fec785f43c4a3bacedecd8472dd P a0e901103604414c829b40edf9f24c28 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:13:17.861301 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:13:17.877386 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:17.908600 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:17.917912 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:17.949826 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 14:13:17.950692 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 14:13:17.962888 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:17.992826 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:18.091361 32652 raft_consensus.cc:998] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.050s)
I20240501 14:13:18.092294 32652 consensus_queue.cc:237] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:18.094738 28743 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:13:18.095469 32715 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:13:18.098656 32652 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:13:18.099325 28743 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:13:18.106487 28595 catalog_manager.cc:5484] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
W20240501 14:13:18.128413 28590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 14:13:18.131904 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e821b93aa9b84ff3a968860a5b33b918: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:18.136071 28874 consensus_queue.cc:237] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:18.141336 28743 raft_consensus.cc:1261] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:13:18.142148 32715 consensus_queue.cc:1035] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:13:18.142725 28809 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Couldn't send request to peer 3a3972b170c54344a4ba61f176f837ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e821b93aa9b84ff3a968860a5b33b918. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:18.146399 32715 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.151707 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e821b93aa9b84ff3a968860a5b33b918 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:13:18.155107 28603 catalog_manager.cc:5484] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.155115 28743 raft_consensus.cc:2932] T e821b93aa9b84ff3a968860a5b33b918 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.189759 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:18.196017 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:18.202844 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.206599 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:18.207037 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:18.265973 32333 raft_consensus.cc:998] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 26acba62745c48e3ba8180748d55b211 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 26acba62745c48e3ba8180748d55b211 for more than 300 seconds (300.263s)
I20240501 14:13:18.266717 32333 consensus_queue.cc:237] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
W20240501 14:13:18.271718 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:13:18.281289 28743 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:13:18.282529 32733 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:13:18.296804 32733 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
W20240501 14:13:18.297818 28810 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:13:18.298522 28743 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:18.306661 28603 catalog_manager.cc:5484] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 7 to 8, VOTER 26acba62745c48e3ba8180748d55b211 (127.16.36.133) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:18.320797 29007 consensus_queue.cc:237] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:18.323010 28743 raft_consensus.cc:1261] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:13:18.324342 32733 consensus_queue.cc:1035] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:13:18.327679 28940 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Couldn't send request to peer 3a3972b170c54344a4ba61f176f837ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb04ec77da194832aa63d69aba307de3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:18.327378 32333 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.328128 28743 raft_consensus.cc:2932] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.332741 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fb04ec77da194832aa63d69aba307de3 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:13:18.335469 28603 catalog_manager.cc:5484] T fb04ec77da194832aa63d69aba307de3 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.467653 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:18.485551 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:18.491357 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:18.508582 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.550025 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:18.605525 28679 consensus_peers.cc:462] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:18.607125 31365 raft_consensus.cc:998] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Attempting to remove follower 8fc5d8222c164a679e7a99966da1eaaf from the Raft config. Reason: Leader has been unable to successfully communicate with peer 8fc5d8222c164a679e7a99966da1eaaf for more than 300 seconds (300.104s)
I20240501 14:13:18.607903 31365 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:13:18.618041 29007 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:13:18.618924 333 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:13:18.621903 31365 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:18.628523 28596 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 7 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:18.629326 29007 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:18.645442 28743 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
W20240501 14:13:18.647830 28678 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827): Couldn't send request to peer 3a3972b170c54344a4ba61f176f837ba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b473e10a8ad4bd58bb44aabd9656e08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:18.648306 29007 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:13:18.648896 333 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:13:18.651940 332 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.655292 29006 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.657949 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9b473e10a8ad4bd58bb44aabd9656e08 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:13:18.660502 28595 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 8 to 9, VOTER 3a3972b170c54344a4ba61f176f837ba (127.16.36.132) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:18.774574 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:18.777885 367 ts_tablet_manager.cc:902] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:13:18.780067 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:18.783311 369 ts_tablet_manager.cc:902] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:13:18.794819 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:18.803102 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:18.819448 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:18.827231 369 tablet_copy_client.cc:308] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:13:18.828246 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet e821b93aa9b84ff3a968860a5b33b918 from peer 3a3972b170c54344a4ba61f176f837ba ({username='slave'} at 127.16.36.132:45291)
I20240501 14:13:18.828416 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet e821b93aa9b84ff3a968860a5b33b918 from peer 3a3972b170c54344a4ba61f176f837ba at {username='slave'} at 127.16.36.132:45291: session id = 3a3972b170c54344a4ba61f176f837ba-e821b93aa9b84ff3a968860a5b33b918
I20240501 14:13:18.834259 28896 tablet_copy_source_session.cc:205] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:13:18.836184 369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e821b93aa9b84ff3a968860a5b33b918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:18.838742 367 tablet_copy_client.cc:308] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:13:18.839850 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet fb04ec77da194832aa63d69aba307de3 from peer 3a3972b170c54344a4ba61f176f837ba ({username='slave'} at 127.16.36.132:37815)
I20240501 14:13:18.840016 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet fb04ec77da194832aa63d69aba307de3 from peer 3a3972b170c54344a4ba61f176f837ba at {username='slave'} at 127.16.36.132:37815: session id = 3a3972b170c54344a4ba61f176f837ba-fb04ec77da194832aa63d69aba307de3
I20240501 14:13:18.844552 29027 tablet_copy_source_session.cc:205] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:13:18.847911 367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb04ec77da194832aa63d69aba307de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:18.861501 369 tablet_copy_client.cc:783] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 5 data blocks...
I20240501 14:13:18.882069 369 tablet_copy_client.cc:647] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:13:18.884093 369 tablet_copy_client.cc:523] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:13:18.886142 367 tablet_copy_client.cc:783] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 5 data blocks...
I20240501 14:13:18.893272 367 tablet_copy_client.cc:647] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:13:18.895373 367 tablet_copy_client.cc:523] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:13:18.895545 369 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:13:18.907887 367 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:13:18.942405 369 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:13:18.943130 369 tablet_bootstrap.cc:492] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap complete.
I20240501 14:13:18.943499 369 ts_tablet_manager.cc:1367] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.048s user 0.029s sys 0.014s
I20240501 14:13:18.944545 369 raft_consensus.cc:348] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:18.944965 369 raft_consensus.cc:729] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:13:18.945242 369 consensus_queue.cc:260] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:18.945837 369 ts_tablet_manager.cc:1398] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:13:18.947484 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 3a3972b170c54344a4ba61f176f837ba-e821b93aa9b84ff3a968860a5b33b918 received from {username='slave'} at 127.16.36.132:45291
I20240501 14:13:18.947618 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 3a3972b170c54344a4ba61f176f837ba-e821b93aa9b84ff3a968860a5b33b918 on tablet e821b93aa9b84ff3a968860a5b33b918 with peer 3a3972b170c54344a4ba61f176f837ba
I20240501 14:13:18.953611 367 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:13:18.954821 367 tablet_bootstrap.cc:492] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap complete.
I20240501 14:13:18.955132 367 ts_tablet_manager.cc:1367] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.017s
I20240501 14:13:18.956375 367 raft_consensus.cc:348] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:18.956626 367 raft_consensus.cc:729] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
W20240501 14:13:18.956969 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240501 14:13:18.956925 367 consensus_queue.cc:260] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:18.958469 367 ts_tablet_manager.cc:1398] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:13:18.959265 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session 3a3972b170c54344a4ba61f176f837ba-fb04ec77da194832aa63d69aba307de3 received from {username='slave'} at 127.16.36.132:37815
I20240501 14:13:18.959386 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session 3a3972b170c54344a4ba61f176f837ba-fb04ec77da194832aa63d69aba307de3 on tablet fb04ec77da194832aa63d69aba307de3 with peer 3a3972b170c54344a4ba61f176f837ba
I20240501 14:13:19.086356 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:19.093741 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:19.104789 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:19.107487 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.108902 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:19.187036 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(e821b93aa9b84ff3a968860a5b33b918): 4921 bytes on disk
I20240501 14:13:19.187774 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(e821b93aa9b84ff3a968860a5b33b918) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 14:13:19.189234 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(fb04ec77da194832aa63d69aba307de3): 4834 bytes on disk
I20240501 14:13:19.189896 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(fb04ec77da194832aa63d69aba307de3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 14:13:19.196813 29138 raft_consensus.cc:1203] T fb04ec77da194832aa63d69aba307de3 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:13:19.216631 29138 raft_consensus.cc:1203] T e821b93aa9b84ff3a968860a5b33b918 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:13:19.239619 32333 raft_consensus.cc:998] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 8fc5d8222c164a679e7a99966da1eaaf from the Raft config. Reason: Leader has been unable to successfully communicate with peer 8fc5d8222c164a679e7a99966da1eaaf for more than 300 seconds (300.357s)
I20240501 14:13:19.242085 32333 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:19.250559 29138 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:13:19.251114 367 ts_tablet_manager.cc:902] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:13:19.251245 32333 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:13:19.252444 367 tablet_copy_client.cc:308] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:13:19.253822 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 9b473e10a8ad4bd58bb44aabd9656e08 from peer 3a3972b170c54344a4ba61f176f837ba ({username='slave'} at 127.16.36.132:45721)
I20240501 14:13:19.253993 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 9b473e10a8ad4bd58bb44aabd9656e08 from peer 3a3972b170c54344a4ba61f176f837ba at {username='slave'} at 127.16.36.132:45721: session id = 3a3972b170c54344a4ba61f176f837ba-9b473e10a8ad4bd58bb44aabd9656e08
I20240501 14:13:19.255549 437 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:19.257884 28765 tablet_copy_source_session.cc:205] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:13:19.259493 367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b473e10a8ad4bd58bb44aabd9656e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:19.259804 29136 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:19.262457 28603 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:19.271545 367 tablet_copy_client.cc:783] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 5 data blocks...
I20240501 14:13:19.279130 367 tablet_copy_client.cc:647] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Starting download of 1 WAL segments...
W20240501 14:13:19.280524 28589 catalog_manager.cc:4573] TS 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b473e10a8ad4bd58bb44aabd9656e08: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:19.281028 367 tablet_copy_client.cc:523] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:13:19.285571 367 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap starting.
I20240501 14:13:19.290174 29006 consensus_queue.cc:237] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:13:19.292583 29136 raft_consensus.cc:1261] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 14:13:19.292848 28942 consensus_peers.cc:485] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efe7fed8e5d44e8a80ffbdc27619a74d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:19.293313 437 consensus_queue.cc:1035] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:13:19.296943 436 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.297694 29136 raft_consensus.cc:2932] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.302481 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet efe7fed8e5d44e8a80ffbdc27619a74d with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 14:13:19.304692 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
I20240501 14:13:19.304615 28595 catalog_manager.cc:5484] T efe7fed8e5d44e8a80ffbdc27619a74d P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 10 to 11, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.336789 367 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:13:19.337265 367 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: Bootstrap complete.
I20240501 14:13:19.337554 367 ts_tablet_manager.cc:1367] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: Time spent bootstrapping tablet: real 0.052s user 0.032s sys 0.001s
I20240501 14:13:19.338183 435 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:13:19.338366 435 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:13:19.339247 435 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:13:19.341164 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
I20240501 14:13:19.343163 367 raft_consensus.cc:348] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:19.343497 367 raft_consensus.cc:729] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3972b170c54344a4ba61f176f837ba, State: Initialized, Role: FOLLOWER
I20240501 14:13:19.343923 367 consensus_queue.cc:260] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:19.344784 367 ts_tablet_manager.cc:1398] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20240501 14:13:19.345675 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session 3a3972b170c54344a4ba61f176f837ba-9b473e10a8ad4bd58bb44aabd9656e08 received from {username='slave'} at 127.16.36.132:45721
I20240501 14:13:19.345836 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session 3a3972b170c54344a4ba61f176f837ba-9b473e10a8ad4bd58bb44aabd9656e08 on tablet 9b473e10a8ad4bd58bb44aabd9656e08 with peer 3a3972b170c54344a4ba61f176f837ba
W20240501 14:13:19.353613 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:19.353780 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:13:19.354043 435 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:19.399694 462 raft_consensus.cc:998] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba: Attempting to remove follower 8fc5d8222c164a679e7a99966da1eaaf from the Raft config. Reason: Leader has been unable to successfully communicate with peer 8fc5d8222c164a679e7a99966da1eaaf for more than 300 seconds (300.311s)
I20240501 14:13:19.400589 462 consensus_queue.cc:237] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:13:19.403164 29006 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:13:19.403846 465 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:13:19.414815 430 raft_consensus.cc:998] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094: Attempting to remove follower 8fc5d8222c164a679e7a99966da1eaaf from the Raft config. Reason: Leader has been unable to successfully communicate with peer 8fc5d8222c164a679e7a99966da1eaaf for more than 300 seconds (300.517s)
I20240501 14:13:19.415940 430 consensus_queue.cc:237] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:13:19.422506 465 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:13:19.427597 29007 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:13:19.434518 28603 catalog_manager.cc:5484] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 7 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:13:19.442684 29184 maintenance_manager.cc:419] P 3a3972b170c54344a4ba61f176f837ba: Scheduling UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08): 4834 bytes on disk
I20240501 14:13:19.443495 29075 maintenance_manager.cc:643] P 3a3972b170c54344a4ba61f176f837ba: UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 14:13:19.461623 29136 consensus_queue.cc:237] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:13:19.465112 29006 raft_consensus.cc:1261] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3972b170c54344a4ba61f176f837ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:13:19.465654 29073 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b6a54ac32694640bffdf1c7f3b035cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:19.465946 462 consensus_queue.cc:1035] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:13:19.468370 28743 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:13:19.469839 462 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.475318 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6b6a54ac32694640bffdf1c7f3b035cc with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:13:19.475592 29006 raft_consensus.cc:2932] T 6b6a54ac32694640bffdf1c7f3b035cc P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.477351 28597 catalog_manager.cc:5484] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 8 to 9, VOTER a0e901103604414c829b40edf9f24c28 (127.16.36.135) added. New cstate: current_term: 1 leader_uuid: "3a3972b170c54344a4ba61f176f837ba" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.482700 430 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:13:19.483377 30925 raft_consensus.cc:1261] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:13:19.490196 435 consensus_queue.cc:1035] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:13:19.496871 435 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.504058 30925 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.509425 28744 raft_consensus.cc:2932] T 2998b60071cd408aa1cd06fb57d76a8f P 343a85ba3b514663af5320cd14a9e144 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.515053 28597 catalog_manager.cc:5484] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 6 to 7, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.530562 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:19.549643 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:19.556779 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:19.586035 28679 consensus_peers.cc:485] T 763b2e84afa64924ad5eeeeb3265a1d8 P 343a85ba3b514663af5320cd14a9e144 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:13:19.595623 30907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:13:19.600795 430 raft_consensus.cc:998] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Attempting to remove follower 8fc5d8222c164a679e7a99966da1eaaf from the Raft config. Reason: Leader has been unable to successfully communicate with peer 8fc5d8222c164a679e7a99966da1eaaf for more than 300 seconds (300.411s)
I20240501 14:13:19.601564 430 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:13:19.609668 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:19.614807 30925 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:13:19.615872 430 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:13:19.616508 28941 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:13:19.641539 32715 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.642522 30926 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.649152 28602 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 reported cstate change: config changed from index 9 to 10, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } }
I20240501 14:13:19.654903 28875 consensus_queue.cc:237] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
W20240501 14:13:19.657847 28810 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): Couldn't send request to peer 7e7bc120bd74479ab378f779778fd1e8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6feda9b53e484e7ea40a5890b7d51ac4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:19.658397 30926 raft_consensus.cc:1261] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Refusing update from remote peer cd3c04ebb6c84b15a1b3adc748737094: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:13:19.659091 32715 consensus_queue.cc:1035] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:13:19.662950 430 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:13:19.671329 30927 raft_consensus.cc:2932] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
W20240501 14:13:19.680495 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:13:19.695405 28602 catalog_manager.cc:5484] T 6feda9b53e484e7ea40a5890b7d51ac4 P a0e901103604414c829b40edf9f24c28 reported cstate change: config changed from index 10 to 11, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) added. New cstate: current_term: 1 leader_uuid: "cd3c04ebb6c84b15a1b3adc748737094" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } }
I20240501 14:13:19.704546 28588 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6feda9b53e484e7ea40a5890b7d51ac4 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:13:19.716117 29136 raft_consensus.cc:1203] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:13:19.723389 505 ts_tablet_manager.cc:902] T efe7fed8e5d44e8a80ffbdc27619a74d P a0e901103604414c829b40edf9f24c28: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
F20240501 14:13:19.723810 505 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-12510f71b6a742ab8cc359243de8119b.queue_time_us, tablet-download-pool-12510f71b6a742ab8cc359243de8119b.queue_time_us) (tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_cpu_time_us, tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_cpu_time_us) (tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_wall_time_us, tablet-download-pool-12510f71b6a742ab8cc359243de8119b.run_wall_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.queue_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.queue_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_cpu_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_cpu_time_us) (tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_wall_time_us, tablet-download-pool-1fdedd39e12a49659d989f37b3287d87.run_wall_time_us) (tablet-download-pool-2020fe37059a4e74bb45190398e4e177.queue_time_us, tablet-download-pool-2020fe37059a4e74bb45190398e4e177.queue_time_us) (tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_cpu_time_us, tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_cpu_time_us) (tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_wall_time_us, tablet-download-pool-2020fe37059a4e74bb45190398e4e177.run_wall_time_us) (tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.queue_time_us, tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.queue_time_us) (tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_cpu_time_us, tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_cpu_time_us) (tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_wall_time_us, tablet-download-pool-2998b60071cd408aa1cd06fb57d76a8f.run_wall_time_us) (tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.queue_time_us, tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.queue_time_us) (tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_cpu_time_us, tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_cpu_time_us) (tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_wall_time_us, tablet-download-pool-60b3db79ff754e8bbe9e5865f7104312.run_wall_time_us) (tablet-download-pool-6aaf52ee71b44511b7b8909711ab0a42.queue_time_us, tablet-download-pool-6aaf52ee71b44511b7b8909711ab0a42.queue_time_us) (tablet-download-pool-6aaf52ee71b44511b7b8909711ab0a42.run_cpu_time_us, tablet-download-pool-6aaf52ee71b44511b7b8909711ab0a42.run_cpu_time_us) (tablet-download-pool-6aaf52ee71b44511b7b8909711ab0a42.run_wall_time_us, tablet-download-pool-6aaf52ee71b44511b7b8909711ab0a42.run_wall_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.queue_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.queue_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_cpu_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_cpu_time_us) (tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_wall_time_us, tablet-download-pool-6feda9b53e484e7ea40a5890b7d51ac4.run_wall_time_us) (tablet-download-pool-7c0bc741ed064dd7ad8ec70219496c6f.queue_time_us, tablet-download-pool-7c0bc741ed064dd7ad8ec70219496c6f.queue_time_us) (tablet-download-pool-7c0bc741ed064dd7ad8ec70219496c6f.run_cpu_time_us, tablet-download-pool-7c0bc741ed064dd7ad8ec70219496c6f.run_cpu_time_us) (tablet-download-pool-7c0bc741ed064dd7ad8ec70219496c6f.run_wall_time_us, tablet-download-pool-7c0bc741ed064dd7ad8ec70219496c6f.run_wall_time_us) (tablet-download-pool-86bfedcdabe44e7ab135674163935786.queue_time_us, tablet-download-pool-86bfedcdabe44e7ab135674163935786.queue_time_us) (tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_cpu_time_us, tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_cpu_time_us) (tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_wall_time_us, tablet-download-pool-86bfedcdabe44e7ab135674163935786.run_wall_time_us) (tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.queue_time_us, tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.queue_time_us) (tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_cpu_time_us, tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_cpu_time_us) (tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_wall_time_us, tablet-download-pool-de161b10c84f4bcc85f8cf7b199f99db.run_wall_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.queue_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.queue_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_cpu_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_cpu_time_us) (tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_wall_time_us, tablet-download-pool-e0bf08cbb6d347d783b1fa7762c802df.run_wall_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.queue_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.queue_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_cpu_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_cpu_time_us) (tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_wall_time_us, tablet-download-pool-e8b9db597e654ed48b679abf58b8a2c8.run_wall_time_us) (tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.queue_time_us, tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.queue_time_us) (tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_cpu_time_us, tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_cpu_time_us) (tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_wall_time_us, tablet-download-pool-ec1b3fec785f43c4a3bacedecd8472dd.run_wall_time_us) (tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.queue_time_us, tablet-download-pool-efe7fed8e5d44e8a80ffbdc27619a74d.queue_time_us) (tablet-download-pool-f516f01e242d4b8789130201f748f3ea.queue_time_us, tablet-download-pool-f516f01e242d4b8789130201f748f3ea.queue_time_us) (tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_cpu_time_us, tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_cpu_time_us) (tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_wall_time_us, tablet-download-pool-f516f01e242d4b8789130201f748f3ea.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f632164bdcd google::LogMessage::Fail() at ??:0
@ 0x7f632164fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f632164b7cc google::LogMessage::Flush() at ??:0
@ 0x7f632164cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f632438c742 kudu::TraceMetrics::InternName() at ??:0
W20240501 14:13:19.760659 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
@ 0x7f6324358f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6324352cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f63337a58de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f63337bddaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f63338e8480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f6333920a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f6333903981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f633390370e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f632435cb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f632436019c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6324360019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f632433b01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f63276c06db start_thread at ??:0
@ 0x7f63204fe71f clone at ??:0
I20240501 14:13:19.919364 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:19.937150 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:19.937218 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:19.959859 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:20.063310 559 ts_tablet_manager.cc:902] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: Initiating tablet copy from peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
I20240501 14:13:20.085927 559 tablet_copy_client.cc:308] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.130:37325
I20240501 14:13:20.086771 28896 tablet_copy_service.cc:140] P cd3c04ebb6c84b15a1b3adc748737094: Received BeginTabletCopySession request for tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer 7e7bc120bd74479ab378f779778fd1e8 ({username='slave'} at 127.16.36.131:53359)
I20240501 14:13:20.086982 28896 tablet_copy_service.cc:161] P cd3c04ebb6c84b15a1b3adc748737094: Beginning new tablet copy session on tablet 6feda9b53e484e7ea40a5890b7d51ac4 from peer 7e7bc120bd74479ab378f779778fd1e8 at {username='slave'} at 127.16.36.131:53359: session id = 7e7bc120bd74479ab378f779778fd1e8-6feda9b53e484e7ea40a5890b7d51ac4
I20240501 14:13:20.091118 28896 tablet_copy_source_session.cc:205] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:13:20.092753 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6feda9b53e484e7ea40a5890b7d51ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:20.100926 559 tablet_copy_client.cc:783] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Starting download of 5 data blocks...
I20240501 14:13:20.110987 559 tablet_copy_client.cc:647] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:13:20.114606 559 tablet_copy_client.cc:523] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:13:20.119302 559 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap starting.
W20240501 14:13:20.144121 28679 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Couldn't send request to peer 826ccf8358b6453d88bcb132290b82a5. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240501 14:13:20.159222 559 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:13:20.159715 559 tablet_bootstrap.cc:492] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: Bootstrap complete.
I20240501 14:13:20.160099 559 ts_tablet_manager.cc:1367] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.000s
I20240501 14:13:20.161113 559 raft_consensus.cc:348] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:13:20.161398 559 raft_consensus.cc:729] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Initialized, Role: FOLLOWER
I20240501 14:13:20.161896 559 consensus_queue.cc:260] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } }
I20240501 14:13:20.162672 559 ts_tablet_manager.cc:1398] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:13:20.163547 28896 tablet_copy_service.cc:342] P cd3c04ebb6c84b15a1b3adc748737094: Request end of tablet copy session 7e7bc120bd74479ab378f779778fd1e8-6feda9b53e484e7ea40a5890b7d51ac4 received from {username='slave'} at 127.16.36.131:53359
I20240501 14:13:20.163692 28896 tablet_copy_service.cc:433] P cd3c04ebb6c84b15a1b3adc748737094: ending tablet copy session 7e7bc120bd74479ab378f779778fd1e8-6feda9b53e484e7ea40a5890b7d51ac4 on tablet 6feda9b53e484e7ea40a5890b7d51ac4 with peer 7e7bc120bd74479ab378f779778fd1e8
I20240501 14:13:20.186765 464 raft_consensus.cc:482] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:20.187065 464 raft_consensus.cc:504] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:13:20.188130 464 leader_election.cc:288] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601)
W20240501 14:13:20.189986 29072 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:20.202409 29053 maintenance_manager.cc:419] P 7e7bc120bd74479ab378f779778fd1e8: Scheduling UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4): 4921 bytes on disk
I20240501 14:13:20.203284 28944 maintenance_manager.cc:643] P 7e7bc120bd74479ab378f779778fd1e8: UndoDeltaBlockGCOp(6feda9b53e484e7ea40a5890b7d51ac4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
W20240501 14:13:20.206142 29073 leader_election.cc:334] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:20.206317 29073 leader_election.cc:302] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:20.206672 464 raft_consensus.cc:2726] T 4d8c435795a04b1b90b6e2ff914676c8 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:20.209296 28811 connection.cc:535] client connection to 127.16.36.135:42127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:13:20.209357 28680 connection.cc:535] client connection to 127.16.36.135:42127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:13:20.209551 28942 connection.cc:535] client connection to 127.16.36.135:42127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:13:20.209771 28942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 26 similar messages]
W20240501 14:13:20.211171 28811 consensus_peers.cc:485] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.211359 28811 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.211467 28811 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.211585 28811 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.212281 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:20.212424 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.212368 28680 consensus_peers.cc:485] T 6aaf52ee71b44511b7b8909711ab0a42 P 343a85ba3b514663af5320cd14a9e144 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.216071 29073 connection.cc:698] client connection to 127.16.36.135:42127 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240501 14:13:20.218941 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:20.219136 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.219269 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:20.244589 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:20.270730 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:20.288129 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 14:13:20.301129 28589 catalog_manager.cc:4573] TS 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6feda9b53e484e7ea40a5890b7d51ac4: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:20.304970 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 14:13:20.341504 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:13:20.343438 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:20.348255 28744 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
W20240501 14:13:20.355260 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:20.364848 29136 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:13:20.365553 333 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:13:20.390789 614 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:20.391575 29136 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:20.398998 28597 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba reported cstate change: config changed from index 9 to 10, VOTER 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131) evicted. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
I20240501 14:13:20.413239 28986 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b473e10a8ad4bd58bb44aabd9656e08 with delete_type TABLET_DATA_TOMBSTONED (TS 7e7bc120bd74479ab378f779778fd1e8 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53622
I20240501 14:13:20.413185 28743 consensus_queue.cc:237] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:20.413872 618 tablet_replica.cc:330] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: stopping tablet replica
I20240501 14:13:20.414098 618 raft_consensus.cc:2229] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:13:20.414477 618 raft_consensus.cc:2258] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:13:20.415447 28678 consensus_peers.cc:485] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 -> Peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Couldn't send request to peer cd3c04ebb6c84b15a1b3adc748737094. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:20.415705 618 ts_tablet_manager.cc:1875] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:13:20.416697 29136 raft_consensus.cc:1261] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Refusing update from remote peer 343a85ba3b514663af5320cd14a9e144: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:13:20.417312 333 consensus_queue.cc:1035] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:13:20.421314 624 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:13:20.422103 29138 raft_consensus.cc:2932] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
W20240501 14:13:20.424093 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:20.426553 28590 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9b473e10a8ad4bd58bb44aabd9656e08 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:13:20.429313 28597 catalog_manager.cc:5484] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144 reported cstate change: config changed from index 10 to 11, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New cstate: current_term: 1 leader_uuid: "343a85ba3b514663af5320cd14a9e144" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:13:20.442047 618 ts_tablet_manager.cc:1888] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:13:20.442215 618 log.cc:1196] T 9b473e10a8ad4bd58bb44aabd9656e08 P 7e7bc120bd74479ab378f779778fd1e8: Deleting WAL directory at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b473e10a8ad4bd58bb44aabd9656e08
I20240501 14:13:20.443179 28589 catalog_manager.cc:4830] TS 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): tablet 9b473e10a8ad4bd58bb44aabd9656e08 (table rebalance_test_table_9 [id=e018134a87f14b58a0ec6e721995eae7]) successfully deleted
W20240501 14:13:20.493909 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111) [suppressed 36 similar messages]
I20240501 14:13:20.613289 29006 raft_consensus.cc:1203] T 6feda9b53e484e7ea40a5890b7d51ac4 P 7e7bc120bd74479ab378f779778fd1e8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240501 14:13:20.731204 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:20.780973 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:20.806352 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:20.812278 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:13:20.842301 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:13:20.870848 581 raft_consensus.cc:482] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:20.871080 581 raft_consensus.cc:504] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } }
I20240501 14:13:20.872025 581 leader_election.cc:288] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:13:20.874079 29072 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:20.892554 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:20.892751 29073 leader_election.cc:302] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 26acba62745c48e3ba8180748d55b211, 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:13:20.893569 581 raft_consensus.cc:2726] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:20.893826 29073 leader_election.cc:334] T 1fdedd39e12a49659d989f37b3287d87 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:20.894577 704 ts_tablet_manager.cc:902] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Initiating tablet copy from peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051)
I20240501 14:13:20.929597 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:20.943681 704 tablet_copy_client.cc:308] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.129:41051
I20240501 14:13:20.944694 28765 tablet_copy_service.cc:140] P 343a85ba3b514663af5320cd14a9e144: Received BeginTabletCopySession request for tablet 9b473e10a8ad4bd58bb44aabd9656e08 from peer cd3c04ebb6c84b15a1b3adc748737094 ({username='slave'} at 127.16.36.130:44247)
I20240501 14:13:20.944902 28765 tablet_copy_service.cc:161] P 343a85ba3b514663af5320cd14a9e144: Beginning new tablet copy session on tablet 9b473e10a8ad4bd58bb44aabd9656e08 from peer cd3c04ebb6c84b15a1b3adc748737094 at {username='slave'} at 127.16.36.130:44247: session id = cd3c04ebb6c84b15a1b3adc748737094-9b473e10a8ad4bd58bb44aabd9656e08
I20240501 14:13:20.948781 28765 tablet_copy_source_session.cc:205] T 9b473e10a8ad4bd58bb44aabd9656e08 P 343a85ba3b514663af5320cd14a9e144: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:13:20.950599 704 ts_tablet_manager.cc:1875] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:13:20.959307 704 ts_tablet_manager.cc:1888] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:13:20.959764 704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b473e10a8ad4bd58bb44aabd9656e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:20.964033 704 tablet_copy_client.cc:783] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 5 data blocks...
I20240501 14:13:20.970377 704 tablet_copy_client.cc:647] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 1 WAL segments...
I20240501 14:13:20.972316 704 tablet_copy_client.cc:523] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:13:20.977212 704 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:13:20.983850 581 raft_consensus.cc:482] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0e901103604414c829b40edf9f24c28)
I20240501 14:13:20.984127 581 raft_consensus.cc:504] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:20.985138 581 leader_election.cc:288] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127), 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:13:20.986532 29006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "3a3972b170c54344a4ba61f176f837ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "7e7bc120bd74479ab378f779778fd1e8" is_pre_election: true
W20240501 14:13:20.993409 29073 leader_election.cc:334] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:20.993600 29073 leader_election.cc:302] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 7e7bc120bd74479ab378f779778fd1e8, a0e901103604414c829b40edf9f24c28
I20240501 14:13:20.994124 581 raft_consensus.cc:2726] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:21.005781 614 raft_consensus.cc:482] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a0e901103604414c829b40edf9f24c28)
I20240501 14:13:21.006080 614 raft_consensus.cc:504] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:13:21.007138 614 leader_election.cc:288] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:13:21.008602 28679 leader_election.cc:334] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:21.009076 28680 leader_election.cc:334] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:21.009250 28680 leader_election.cc:302] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 343a85ba3b514663af5320cd14a9e144; no voters: 8fc5d8222c164a679e7a99966da1eaaf, a0e901103604414c829b40edf9f24c28
I20240501 14:13:21.009958 614 raft_consensus.cc:2726] T ec1b3fec785f43c4a3bacedecd8472dd P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 14:13:21.042785 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:21.047228 704 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:13:21.047669 704 tablet_bootstrap.cc:492] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap complete.
I20240501 14:13:21.047996 704 ts_tablet_manager.cc:1367] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.071s user 0.060s sys 0.007s
I20240501 14:13:21.048815 704 raft_consensus.cc:348] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:21.049052 704 raft_consensus.cc:729] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:13:21.049340 704 consensus_queue.cc:260] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:21.051350 704 ts_tablet_manager.cc:1398] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:21.052083 28765 tablet_copy_service.cc:342] P 343a85ba3b514663af5320cd14a9e144: Request end of tablet copy session cd3c04ebb6c84b15a1b3adc748737094-9b473e10a8ad4bd58bb44aabd9656e08 received from {username='slave'} at 127.16.36.130:44247
I20240501 14:13:21.052232 28765 tablet_copy_service.cc:433] P 343a85ba3b514663af5320cd14a9e144: ending tablet copy session cd3c04ebb6c84b15a1b3adc748737094-9b473e10a8ad4bd58bb44aabd9656e08 on tablet 9b473e10a8ad4bd58bb44aabd9656e08 with peer cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:13:21.066884 28723 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:21.087461 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08): 4834 bytes on disk
I20240501 14:13:21.088532 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(9b473e10a8ad4bd58bb44aabd9656e08) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
I20240501 14:13:21.094686 29118 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:13:21.097002 700 raft_consensus.cc:998] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8: Attempting to remove follower 8fc5d8222c164a679e7a99966da1eaaf from the Raft config. Reason: Leader has been unable to successfully communicate with peer 8fc5d8222c164a679e7a99966da1eaaf for more than 300 seconds (300.376s)
I20240501 14:13:21.097918 700 consensus_queue.cc:237] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } }
I20240501 14:13:21.135154 28856 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:13:21.157773 28743 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 14:13:21.157908 28986 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:21.158608 700 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:13:21.162129 700 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:21.167757 28743 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:21.173139 28602 catalog_manager.cc:5484] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 6 to 8, VOTER 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134) evicted. New cstate: current_term: 3 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } }
I20240501 14:13:21.201455 29006 consensus_queue.cc:237] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:21.213353 28743 raft_consensus.cc:1261] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 14:13:21.214242 437 consensus_queue.cc:1035] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:13:21.214591 28940 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Couldn't send request to peer cd3c04ebb6c84b15a1b3adc748737094. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39f1ab3a94db41b08ac90bba52418a93. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:21.216835 700 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:13:21.222041 28743 raft_consensus.cc:2932] T 39f1ab3a94db41b08ac90bba52418a93 P 343a85ba3b514663af5320cd14a9e144 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:13:21.226059 28589 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 39f1ab3a94db41b08ac90bba52418a93 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 14:13:21.228577 28603 catalog_manager.cc:5484] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: config changed from index 8 to 9, VOTER cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130) added. New cstate: current_term: 3 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } }
I20240501 14:13:21.271548 700 raft_consensus.cc:482] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a0e901103604414c829b40edf9f24c28)
I20240501 14:13:21.271867 700 raft_consensus.cc:504] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:21.273116 700 leader_election.cc:288] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:13:21.273214 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba" is_pre_election: true
I20240501 14:13:21.273481 29138 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 2.
I20240501 14:13:21.273983 28940 leader_election.cc:302] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:13:21.274540 700 raft_consensus.cc:2781] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:13:21.274753 700 raft_consensus.cc:482] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a0e901103604414c829b40edf9f24c28)
I20240501 14:13:21.274870 700 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:13:21.275036 28942 leader_election.cc:334] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:21.277541 700 raft_consensus.cc:504] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:21.278591 700 leader_election.cc:288] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 election: Requested vote from peers a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127), 3a3972b170c54344a4ba61f176f837ba (127.16.36.132:42827)
I20240501 14:13:21.278743 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b3db79ff754e8bbe9e5865f7104312" candidate_uuid: "7e7bc120bd74479ab378f779778fd1e8" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3a3972b170c54344a4ba61f176f837ba"
I20240501 14:13:21.278957 29138 raft_consensus.cc:3037] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:13:21.281505 29138 raft_consensus.cc:2454] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e7bc120bd74479ab378f779778fd1e8 in term 3.
I20240501 14:13:21.281960 28940 leader_election.cc:302] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba, 7e7bc120bd74479ab378f779778fd1e8; no voters:
I20240501 14:13:21.282825 700 raft_consensus.cc:2781] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 FOLLOWER]: Leader election won for term 3
W20240501 14:13:21.283001 28942 leader_election.cc:334] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:21.283394 700 raft_consensus.cc:686] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [term 3 LEADER]: Becoming Leader. State: Replica: 7e7bc120bd74479ab378f779778fd1e8, State: Running, Role: LEADER
I20240501 14:13:21.283679 700 consensus_queue.cc:237] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:21.295408 28602 catalog_manager.cc:5484] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 reported cstate change: term changed from 2 to 3, leader changed from a0e901103604414c829b40edf9f24c28 (127.16.36.135) to 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131). New cstate: current_term: 3 leader_uuid: "7e7bc120bd74479ab378f779778fd1e8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } }
W20240501 14:13:21.337595 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:21.349656 28680 consensus_peers.cc:485] T 014635e01a22420b8bc21e879449749b P 343a85ba3b514663af5320cd14a9e144 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240501 14:13:21.396093 29073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:21.400549 28589 catalog_manager.cc:4573] TS 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 39f1ab3a94db41b08ac90bba52418a93: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:21.413246 28875 raft_consensus.cc:1203] T 9b473e10a8ad4bd58bb44aabd9656e08 P cd3c04ebb6c84b15a1b3adc748737094 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240501 14:13:21.463761 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:21.672163 29138 raft_consensus.cc:1261] T 60b3db79ff754e8bbe9e5865f7104312 P 3a3972b170c54344a4ba61f176f837ba [term 3 FOLLOWER]: Refusing update from remote peer 7e7bc120bd74479ab378f779778fd1e8: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 14:13:21.672828 700 consensus_queue.cc:1035] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 14:13:21.685422 28942 consensus_peers.cc:485] T 60b3db79ff754e8bbe9e5865f7104312 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:21.691375 28811 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240501 14:13:21.766211 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:21.777096 31111 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task1g_oHS/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:02:28.230017 31118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:28.230883 31118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:28.246302 31118 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240501 14:02:29.322366 31289 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.16.36.190:41971, user_credentials={real_user=slave}} blocked reactor thread for 49052.6us
I20240501 14:02:31.987716 31118 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:02:33.015586 31783 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.16.36.129:41051, user_credentials={real_user=slave}} blocked reactor thread for 46015.5us
W20240501 14:02:33.515606 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.521612 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.522431 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:33.522873 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.523187 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.523488 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:33.523788 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.524094 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:33.524425 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.524796 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.525099 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:33.525415 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:33.525789 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:33.526082 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:34.038816 31118 rebalancer_tool.cc:1074] tablet 2020fe37059a4e74bb45190398e4e177: '7e7bc120bd74479ab378f779778fd1e8' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:34.554091 31118 rebalancer_tool.cc:1084] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:02:34.554333 31118 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:36.084863 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.085165 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.085299 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.085429 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.085611 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.085757 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:36.085888 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.086000 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.086143 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.086237 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.086344 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:36.086453 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:36.086587 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:36.086706 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
I20240501 14:02:36.653566 31118 rebalancer_tool.cc:1074] tablet fb04ec77da194832aa63d69aba307de3: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:36.835029 31118 rebalancer_tool.cc:1074] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:02:37.197611 31118 rebalancer_tool.cc:1074] tablet de161b10c84f4bcc85f8cf7b199f99db: '3a3972b170c54344a4ba61f176f837ba' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:37.325361 31118 rebalancer_tool.cc:1084] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> '26acba62745c48e3ba8180748d55b211' move ignored: Illegal state: tablet e821b93aa9b84ff3a968860a5b33b918: replica 26acba62745c48e3ba8180748d55b211 is already present
I20240501 14:02:37.325570 31118 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:39.254882 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:39.257195 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.257624 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:39.257879 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.258172 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.258293 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:39.258400 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:39.258539 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.258812 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:39.258957 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:39.259128 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:39.259248 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:39.259361 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:39.643781 31118 rebalancer_tool.cc:1074] tablet f4402de0b48041c383568feb9365147e: '7e7bc120bd74479ab378f779778fd1e8' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:39.825659 31118 rebalancer_tool.cc:1074] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:02:40.917307 31118 rebalancer_tool.cc:1074] tablet 6aaf52ee71b44511b7b8909711ab0a42: 'cd3c04ebb6c84b15a1b3adc748737094' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:41.353749 31118 rebalancer_tool.cc:1074] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:41.861749 31118 rebalancer_tool.cc:1074] tablet bb712628c7764a138f5425d14b724d9e: '7e7bc120bd74479ab378f779778fd1e8' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:42.305696 31118 rebalancer_tool.cc:1074] tablet 1fdedd39e12a49659d989f37b3287d87: 'cd3c04ebb6c84b15a1b3adc748737094' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:42.451314 31118 rebalancer_tool.cc:1074] tablet efe7fed8e5d44e8a80ffbdc27619a74d: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:02:42.852679 31118 rebalancer_tool.cc:1074] tablet e8b9db597e654ed48b679abf58b8a2c8: '343a85ba3b514663af5320cd14a9e144' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:43.389930 31118 rebalancer_tool.cc:1074] tablet 014635e01a22420b8bc21e879449749b: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:43.699092 31118 rebalancer_tool.cc:1084] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '8fc5d8222c164a679e7a99966da1eaaf' move ignored: Illegal state: tablet 763b2e84afa64924ad5eeeeb3265a1d8: replica 8fc5d8222c164a679e7a99966da1eaaf is already present
I20240501 14:02:43.699328 31118 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:45.657043 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:45.657469 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:45.657624 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:45.657771 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:45.657948 31118 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:45.878182 31118 rebalancer_tool.cc:1074] tablet e0bf08cbb6d347d783b1fa7762c802df: '826ccf8358b6453d88bcb132290b82a5' -> '' move scheduled
I20240501 14:02:46.151636 31118 rebalancer_tool.cc:1074] tablet 12510f71b6a742ab8cc359243de8119b: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:02:58.516685 31118 rebalancer_tool.cc:1136] tablet 6aaf52ee71b44511b7b8909711ab0a42: 'cd3c04ebb6c84b15a1b3adc748737094' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:03.618878 31118 rebalancer_tool.cc:1136] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:13.521981 31118 rebalancer_tool.cc:1136] tablet 2020fe37059a4e74bb45190398e4e177: '7e7bc120bd74479ab378f779778fd1e8' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:15.948539 31118 rebalancer_tool.cc:1136] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:16.144173 31118 rebalancer_tool.cc:1074] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:16.384388 31118 rebalancer_tool.cc:1074] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:16.630528 31118 rebalancer_tool.cc:1136] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:16.754789 31118 rebalancer_tool.cc:1136] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:16.802410 31118 rebalancer_tool.cc:1136] tablet e8b9db597e654ed48b679abf58b8a2c8: '343a85ba3b514663af5320cd14a9e144' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:17.085336 31118 rebalancer_tool.cc:1136] tablet 1fdedd39e12a49659d989f37b3287d87: 'cd3c04ebb6c84b15a1b3adc748737094' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:17.187642 31118 rebalancer_tool.cc:1136] tablet 12510f71b6a742ab8cc359243de8119b: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:17.381397 31118 rebalancer_tool.cc:1136] tablet efe7fed8e5d44e8a80ffbdc27619a74d: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:17.445569 31118 rebalancer_tool.cc:1136] tablet 6b6a54ac32694640bffdf1c7f3b035cc: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:17.591334 31118 rebalancer_tool.cc:1136] tablet bb712628c7764a138f5425d14b724d9e: '7e7bc120bd74479ab378f779778fd1e8' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:17.807255 31118 rebalancer_tool.cc:1136] tablet fb04ec77da194832aa63d69aba307de3: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:17.995229 31118 rebalancer_tool.cc:1136] tablet de161b10c84f4bcc85f8cf7b199f99db: '3a3972b170c54344a4ba61f176f837ba' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:18.175585 31118 rebalancer_tool.cc:1136] tablet e0bf08cbb6d347d783b1fa7762c802df: '826ccf8358b6453d88bcb132290b82a5' -> '' move completed: OK
I20240501 14:03:18.325672 31118 rebalancer_tool.cc:1136] tablet 014635e01a22420b8bc21e879449749b: 'cd3c04ebb6c84b15a1b3adc748737094' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:18.499658 31118 rebalancer_tool.cc:1136] tablet f4402de0b48041c383568feb9365147e: '7e7bc120bd74479ab378f779778fd1e8' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:18.814185 31118 rebalancer_tool.cc:1074] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:18.914901 31118 rebalancer_tool.cc:1074] tablet 60b3db79ff754e8bbe9e5865f7104312: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:19.038404 31118 rebalancer_tool.cc:1074] tablet 9b473e10a8ad4bd58bb44aabd9656e08: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:19.218859 31118 rebalancer_tool.cc:1074] tablet e9bb3eaf3f0e40f8902b9426a845d9b2: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:19.296883 31118 rebalancer_tool.cc:1074] tablet f516f01e242d4b8789130201f748f3ea: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:19.378665 31118 rebalancer_tool.cc:1136] tablet f516f01e242d4b8789130201f748f3ea: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:22.989163 31118 rebalancer_tool.cc:1136] tablet 60b3db79ff754e8bbe9e5865f7104312: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move completed: OK
I20240501 14:03:24.572714 31118 rebalancer_tool.cc:1074] tablet ec1b3fec785f43c4a3bacedecd8472dd: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:26.643141 31118 rebalancer_tool.cc:1136] tablet e9bb3eaf3f0e40f8902b9426a845d9b2: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:27.928738 31118 rebalancer_tool.cc:1136] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:27.980434 31118 rebalancer_tool.cc:1136] tablet ec1b3fec785f43c4a3bacedecd8472dd: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
W20240501 14:06:02.642765 30319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.16.36.131:41159, user_credentials={real_user=slave}} blocked reactor thread for 48297.4us
W20240501 14:08:19.712975 16702 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:08:19.311453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.134:44241 (local address 127.0.0.1:50898)
0501 14:08:19.311753 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0501 14:08:19.311774 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 14:08:19.311889 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:08:19.711551 (+399662us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124}
W20240501 14:13:20.210253 541 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:13:19.927624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.135:42127 (local address 127.0.0.1:37760)
0501 14:13:19.927814 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0501 14:13:19.927829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 14:13:19.927927 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:13:20.209946 (+282019us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":119,"spinlock_wait_cycles":5632}
I20240501 14:13:20.610450 31118 rebalancer_tool.cc:1136] tablet 9b473e10a8ad4bd58bb44aabd9656e08: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
Illegal state: tablet server 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): unacceptable health status UNAVAILABLE
I20240501 14:13:21.787284 737 ts_tablet_manager.cc:902] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: Initiating tablet copy from peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159)
I20240501 14:13:21.789070 737 tablet_copy_client.cc:308] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.131:41159
I20240501 14:13:21.789789 29027 tablet_copy_service.cc:140] P 7e7bc120bd74479ab378f779778fd1e8: Received BeginTabletCopySession request for tablet 39f1ab3a94db41b08ac90bba52418a93 from peer cd3c04ebb6c84b15a1b3adc748737094 ({username='slave'} at 127.16.36.130:47511)
I20240501 14:13:21.789937 29027 tablet_copy_service.cc:161] P 7e7bc120bd74479ab378f779778fd1e8: Beginning new tablet copy session on tablet 39f1ab3a94db41b08ac90bba52418a93 from peer cd3c04ebb6c84b15a1b3adc748737094 at {username='slave'} at 127.16.36.130:47511: session id = cd3c04ebb6c84b15a1b3adc748737094-39f1ab3a94db41b08ac90bba52418a93
I20240501 14:13:21.792945 29027 tablet_copy_source_session.cc:205] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:13:21.794548 737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f1ab3a94db41b08ac90bba52418a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:21.802090 737 tablet_copy_client.cc:783] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 5 data blocks...
I20240501 14:13:21.808070 737 tablet_copy_client.cc:647] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Starting download of 1 WAL segments...
I20240501 14:13:21.809921 737 tablet_copy_client.cc:523] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:13:21.815054 737 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap starting.
I20240501 14:13:21.839399 737 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:13:21.839977 737 tablet_bootstrap.cc:492] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: Bootstrap complete.
I20240501 14:13:21.840297 737 ts_tablet_manager.cc:1367] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240501 14:13:21.841120 737 raft_consensus.cc:348] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:21.841372 737 raft_consensus.cc:729] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c04ebb6c84b15a1b3adc748737094, State: Initialized, Role: FOLLOWER
I20240501 14:13:21.841706 737 consensus_queue.cc:260] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:21.843308 737 ts_tablet_manager.cc:1398] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:13:21.845602 28922 maintenance_manager.cc:419] P cd3c04ebb6c84b15a1b3adc748737094: Scheduling UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93): 4834 bytes on disk
I20240501 14:13:21.846156 28813 maintenance_manager.cc:643] P cd3c04ebb6c84b15a1b3adc748737094: UndoDeltaBlockGCOp(39f1ab3a94db41b08ac90bba52418a93) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 14:13:21.851194 29027 tablet_copy_service.cc:342] P 7e7bc120bd74479ab378f779778fd1e8: Request end of tablet copy session cd3c04ebb6c84b15a1b3adc748737094-39f1ab3a94db41b08ac90bba52418a93 received from {username='slave'} at 127.16.36.130:47511
I20240501 14:13:21.851353 29027 tablet_copy_service.cc:433] P 7e7bc120bd74479ab378f779778fd1e8: ending tablet copy session cd3c04ebb6c84b15a1b3adc748737094-39f1ab3a94db41b08ac90bba52418a93 on tablet 39f1ab3a94db41b08ac90bba52418a93 with peer cd3c04ebb6c84b15a1b3adc748737094
W20240501 14:13:21.856869 28810 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:13:21.912175 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:21.919328 31114 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task1g_oHS/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:02:27.647457 31117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:02:27.647909 31117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:02:27.660830 31117 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:02:30.753566 31117 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:02:31.738472 31117 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.508s user 0.124s sys 0.152s
W20240501 14:02:31.738771 31117 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.508s user 0.124s sys 0.152s
W20240501 14:02:32.861356 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.861979 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:32.862167 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.862318 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:32.867311 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:32.867527 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:32.867750 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:32.867946 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:32.868090 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:32.868237 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.868654 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:32.868863 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:32.869130 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:32.869369 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:32.869535 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:33.494374 31117 rebalancer_tool.cc:1084] tablet 60b3db79ff754e8bbe9e5865f7104312: '343a85ba3b514663af5320cd14a9e144' -> 'a0e901103604414c829b40edf9f24c28' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:02:33.499115 31117 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:35.059718 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:35.060106 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:35.060470 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:35.060599 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:35.061501 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:35.061659 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:35.061830 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:35.061950 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:35.062093 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:35.062204 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:35.062330 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:35.062450 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:35.062716 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:35.062832 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:35.062947 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:35.063067 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:35.063192 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
I20240501 14:02:35.631726 31117 rebalancer_tool.cc:1074] tablet f516f01e242d4b8789130201f748f3ea: 'cd3c04ebb6c84b15a1b3adc748737094' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:36.396346 31117 rebalancer_tool.cc:1074] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:36.764254 31117 rebalancer_tool.cc:1084] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> 'a0e901103604414c829b40edf9f24c28' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } }
I20240501 14:02:36.770792 31117 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:38.687791 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:38.688066 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:38.688338 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:38.688764 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:38.688947 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:38.689075 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:38.689306 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:38.689486 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e018134a87f14b58a0ec6e721995eae7: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:38.689605 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:38.689765 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:38.689893 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4ce17c35a7b64e0097eaee91ab51749f: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:38.690033 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server a0e901103604414c829b40edf9f24c28
I20240501 14:02:38.898355 31117 rebalancer_tool.cc:1074] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:02:39.484396 31117 rebalancer_tool.cc:1074] tablet 1fdedd39e12a49659d989f37b3287d87: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:39.893493 31117 rebalancer_tool.cc:1074] tablet ec1b3fec785f43c4a3bacedecd8472dd: '3a3972b170c54344a4ba61f176f837ba' -> 'a0e901103604414c829b40edf9f24c28' move scheduled
I20240501 14:02:40.350509 31117 rebalancer_tool.cc:1074] tablet bb712628c7764a138f5425d14b724d9e: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move scheduled
I20240501 14:02:41.036752 31117 rebalancer_tool.cc:1074] tablet e8b9db597e654ed48b679abf58b8a2c8: '7e7bc120bd74479ab378f779778fd1e8' -> '826ccf8358b6453d88bcb132290b82a5' move scheduled
I20240501 14:02:41.131511 31117 rebalancer_tool.cc:1074] tablet 60b3db79ff754e8bbe9e5865f7104312: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:02:41.495826 31117 rebalancer_tool.cc:1084] tablet 2998b60071cd408aa1cd06fb57d76a8f: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240501 14:02:41.495990 31117 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:43.264376 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:43.264590 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:43.264751 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:43.264959 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7974accec32844bfa75c66b42b48e0eb: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:43.265051 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:43.265151 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6d518ad3360047ccb71449199dc94b10: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server a0e901103604414c829b40edf9f24c28
W20240501 14:02:43.265329 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server 343a85ba3b514663af5320cd14a9e144 to server 8fc5d8222c164a679e7a99966da1eaaf
I20240501 14:02:43.569752 31117 rebalancer_tool.cc:1074] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move scheduled
I20240501 14:02:43.785245 31117 rebalancer_tool.cc:1074] tablet 6aaf52ee71b44511b7b8909711ab0a42: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:02:43.912204 31117 rebalancer_tool.cc:1084] tablet 9b7c7b963588436bb778e45da469c7d5: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move ignored: Illegal state: tablet 9b7c7b963588436bb778e45da469c7d5: replica 26acba62745c48e3ba8180748d55b211 is already present
I20240501 14:02:43.912353 31117 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:02:45.866916 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c026765fe9274ecab686de9c27c27e59: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:45.867107 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 434d653274bc4aa7938133d6f5481332: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:45.867228 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b588a9a3cdf4f31bd439f0eafb9fa7a: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 26acba62745c48e3ba8180748d55b211
W20240501 14:02:45.867484 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 771fb3db7ee841fd8e33c05cb3a9b82e: could not find any suitable replica to move from server cd3c04ebb6c84b15a1b3adc748737094 to server 826ccf8358b6453d88bcb132290b82a5
W20240501 14:02:45.867630 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16d485c2dc3c415b9eb32f71cc8c2632: could not find any suitable replica to move from server 3a3972b170c54344a4ba61f176f837ba to server 8fc5d8222c164a679e7a99966da1eaaf
W20240501 14:02:45.867738 31117 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 42106fd27f0b49b2890a9aaaf268a5e3: could not find any suitable replica to move from server 7e7bc120bd74479ab378f779778fd1e8 to server 826ccf8358b6453d88bcb132290b82a5
I20240501 14:02:45.972461 31117 rebalancer_tool.cc:1074] tablet 014635e01a22420b8bc21e879449749b: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:02:46.138717 31117 rebalancer_tool.cc:1074] tablet de161b10c84f4bcc85f8cf7b199f99db: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:02:53.304987 31117 rebalancer_tool.cc:1136] tablet 6feda9b53e484e7ea40a5890b7d51ac4: '343a85ba3b514663af5320cd14a9e144' -> '826ccf8358b6453d88bcb132290b82a5' move completed: OK
I20240501 14:03:06.449752 31117 rebalancer_tool.cc:1136] tablet 1fdedd39e12a49659d989f37b3287d87: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:17.196615 31117 rebalancer_tool.cc:1136] tablet de161b10c84f4bcc85f8cf7b199f99db: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:17.347195 31117 rebalancer_tool.cc:1136] tablet 014635e01a22420b8bc21e879449749b: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:17.534994 31117 rebalancer_tool.cc:1136] tablet 6aaf52ee71b44511b7b8909711ab0a42: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:17.780211 31117 rebalancer_tool.cc:1136] tablet f516f01e242d4b8789130201f748f3ea: 'cd3c04ebb6c84b15a1b3adc748737094' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
I20240501 14:03:18.114955 31117 rebalancer_tool.cc:1136] tablet e821b93aa9b84ff3a968860a5b33b918: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:18.280807 31117 rebalancer_tool.cc:1136] tablet bb712628c7764a138f5425d14b724d9e: '343a85ba3b514663af5320cd14a9e144' -> '26acba62745c48e3ba8180748d55b211' move completed: OK
W20240501 14:03:18.935396 6351 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:03:18.475919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.136:36399 (local address 127.0.0.1:42628)
0501 14:03:18.580530 (+104611us) negotiation.cc:107] Waiting for socket to connect
0501 14:03:18.580555 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 14:03:18.580735 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:03:18.933950 (+353215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":104459,"thread_start_us":104634,"threads_started":1}
I20240501 14:03:18.936841 31117 rebalancer_tool.cc:1123] tablet e8b9db597e654ed48b679abf58b8a2c8: '7e7bc120bd74479ab378f779778fd1e8' -> '826ccf8358b6453d88bcb132290b82a5' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:03:19.042605 31117 rebalancer_tool.cc:1136] tablet 63b7f2a8d8a84faf86c4de16b3cc6663: 'cd3c04ebb6c84b15a1b3adc748737094' -> '8fc5d8222c164a679e7a99966da1eaaf' move completed: OK
I20240501 14:03:19.153430 31117 rebalancer_tool.cc:1136] tablet 60b3db79ff754e8bbe9e5865f7104312: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:19.295295 31117 rebalancer_tool.cc:1136] tablet ec1b3fec785f43c4a3bacedecd8472dd: '3a3972b170c54344a4ba61f176f837ba' -> 'a0e901103604414c829b40edf9f24c28' move completed: OK
I20240501 14:03:19.375576 31117 rebalancer_tool.cc:1074] tablet efe7fed8e5d44e8a80ffbdc27619a74d: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:03:19.507839 31117 rebalancer_tool.cc:1074] tablet 9b7c7b963588436bb778e45da469c7d5: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move scheduled
I20240501 14:03:19.638684 31117 rebalancer_tool.cc:1074] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:19.815652 31117 rebalancer_tool.cc:1074] tablet e0bf08cbb6d347d783b1fa7762c802df: '826ccf8358b6453d88bcb132290b82a5' -> '' move scheduled
I20240501 14:03:20.101752 31117 rebalancer_tool.cc:1074] tablet 2998b60071cd408aa1cd06fb57d76a8f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:20.303668 31117 rebalancer_tool.cc:1136] tablet 2998b60071cd408aa1cd06fb57d76a8f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:20.346431 31117 rebalancer_tool.cc:1136] tablet e0bf08cbb6d347d783b1fa7762c802df: '826ccf8358b6453d88bcb132290b82a5' -> '' move completed: OK
I20240501 14:03:20.436185 31117 rebalancer_tool.cc:1136] tablet 763b2e84afa64924ad5eeeeb3265a1d8: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
I20240501 14:03:21.688841 31117 rebalancer_tool.cc:1136] tablet efe7fed8e5d44e8a80ffbdc27619a74d: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:21.758764 31117 rebalancer_tool.cc:1074] tablet 2020fe37059a4e74bb45190398e4e177: '3a3972b170c54344a4ba61f176f837ba' -> '' move scheduled
I20240501 14:03:21.823222 31117 rebalancer_tool.cc:1074] tablet 135da52054584a27959efdd9126cfcef: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:03:21.922434 31117 rebalancer_tool.cc:1074] tablet 9b473e10a8ad4bd58bb44aabd9656e08: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:21.973294 31117 rebalancer_tool.cc:1074] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> '' move scheduled
I20240501 14:03:21.997546 31117 rebalancer_tool.cc:1136] tablet 86bfedcdabe44e7ab135674163935786: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:03:28.850827 31117 rebalancer_tool.cc:1123] tablet 9b7c7b963588436bb778e45da469c7d5: 'cd3c04ebb6c84b15a1b3adc748737094' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:03:28.930497 31117 rebalancer_tool.cc:1074] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move scheduled
I20240501 14:03:29.031097 31117 rebalancer_tool.cc:1136] tablet 7c0bc741ed064dd7ad8ec70219496c6f: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
I20240501 14:03:32.469622 31117 rebalancer_tool.cc:1136] tablet 2020fe37059a4e74bb45190398e4e177: '3a3972b170c54344a4ba61f176f837ba' -> '' move completed: OK
W20240501 14:08:19.716601 16764 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:08:19.693332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.134:44241 (local address 127.0.0.1:50902)
0501 14:08:19.693982 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0501 14:08:19.693999 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 14:08:19.694110 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:08:19.716403 (+ 22293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":549}
I20240501 14:08:23.855490 31117 rebalancer_tool.cc:1136] tablet 135da52054584a27959efdd9126cfcef: '343a85ba3b514663af5320cd14a9e144' -> '' move completed: OK
I20240501 14:13:20.683905 31117 rebalancer_tool.cc:1136] tablet 9b473e10a8ad4bd58bb44aabd9656e08: '7e7bc120bd74479ab378f779778fd1e8' -> '' move completed: OK
Illegal state: tablet server 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): unacceptable health status UNAVAILABLE
E20240501 14:13:21.920580 16530 external_mini_cluster.cc:943] Process with UUID 26acba62745c48e3ba8180748d55b211 has crashed
E20240501 14:13:21.920732 16530 external_mini_cluster.cc:943] Process with UUID 8fc5d8222c164a679e7a99966da1eaaf has crashed
E20240501 14:13:21.920818 16530 external_mini_cluster.cc:943] Process with UUID a0e901103604414c829b40edf9f24c28 has crashed
E20240501 14:13:21.920900 16530 external_mini_cluster.cc:943] Process with UUID 826ccf8358b6453d88bcb132290b82a5 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 4
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 14:13:21.922705 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 28663
I20240501 14:13:21.932538 28782 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:13:22.145330 28811 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:13:22.175370 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:22.191337 29072 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:13:22.300072 726 raft_consensus.cc:482] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 26acba62745c48e3ba8180748d55b211)
I20240501 14:13:22.300325 726 raft_consensus.cc:504] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "26acba62745c48e3ba8180748d55b211" member_type: VOTER last_known_addr { host: "127.16.36.133" port: 40601 } } peers { permanent_uuid: "826ccf8358b6453d88bcb132290b82a5" member_type: VOTER last_known_addr { host: "127.16.36.136" port: 36399 } }
I20240501 14:13:22.301334 726 leader_election.cc:288] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601), 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399)
W20240501 14:13:22.303130 28810 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 826ccf8358b6453d88bcb132290b82a5 (127.16.36.136:36399): Network error: Client connection negotiation failed: client connection to 127.16.36.136:36399: connect: Connection refused (error 111)
W20240501 14:13:22.303727 28811 leader_election.cc:334] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
I20240501 14:13:22.303844 28811 leader_election.cc:302] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3c04ebb6c84b15a1b3adc748737094; no voters: 26acba62745c48e3ba8180748d55b211, 826ccf8358b6453d88bcb132290b82a5
I20240501 14:13:22.304155 726 raft_consensus.cc:2726] T 9b7c7b963588436bb778e45da469c7d5 P cd3c04ebb6c84b15a1b3adc748737094 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:22.332979 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:13:22.352152 28875 raft_consensus.cc:1203] T 39f1ab3a94db41b08ac90bba52418a93 P cd3c04ebb6c84b15a1b3adc748737094 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
W20240501 14:13:22.354611 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:22.370329 28811 consensus_peers.cc:485] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:13:22.375918 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 28663
W20240501 14:13:22.385779 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:13:22.436313 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:13:22.438400 29073 connection.cc:535] client connection to 127.16.36.129:41051 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:13:22.438653 28942 connection.cc:535] client connection to 127.16.36.129:41051 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:13:22.439635 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 28794
W20240501 14:13:22.440295 28811 consensus_peers.cc:485] T 2998b60071cd408aa1cd06fb57d76a8f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.440469 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.440475 29073 consensus_peers.cc:485] T f516f01e242d4b8789130201f748f3ea P 3a3972b170c54344a4ba61f176f837ba -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.440608 29073 consensus_peers.cc:485] T 86bfedcdabe44e7ab135674163935786 P 3a3972b170c54344a4ba61f176f837ba -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.440593 28811 consensus_peers.cc:485] T e821b93aa9b84ff3a968860a5b33b918 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.440724 28811 consensus_peers.cc:485] T 7c0bc741ed064dd7ad8ec70219496c6f P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.440966 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.441108 28942 consensus_peers.cc:485] T e9bb3eaf3f0e40f8902b9426a845d9b2 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.441219 28942 consensus_peers.cc:485] T 39f1ab3a94db41b08ac90bba52418a93 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.441330 28942 consensus_peers.cc:485] T 12510f71b6a742ab8cc359243de8119b P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:13:22.441452 28942 consensus_peers.cc:485] T fb04ec77da194832aa63d69aba307de3 P 7e7bc120bd74479ab378f779778fd1e8 -> Peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Couldn't send request to peer 343a85ba3b514663af5320cd14a9e144. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:22.446928 28915 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:13:22.495234 28590 catalog_manager.cc:4573] TS 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e821b93aa9b84ff3a968860a5b33b918: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111)
W20240501 14:13:22.580531 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:22.774456 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:22.890347 28942 consensus_peers.cc:485] T e0bf08cbb6d347d783b1fa7762c802df P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:13:22.957818 28811 consensus_peers.cc:485] T 2020fe37059a4e74bb45190398e4e177 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:13:22.957664 28810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 14:13:22.958345 28811 consensus_peers.cc:485] T 6feda9b53e484e7ea40a5890b7d51ac4 P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:13:22.958752 28811 consensus_peers.cc:485] T de161b10c84f4bcc85f8cf7b199f99db P cd3c04ebb6c84b15a1b3adc748737094 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Couldn't send request to peer a0e901103604414c829b40edf9f24c28. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:13:22.963423 28810 consensus_peers.cc:485] T bb712628c7764a138f5425d14b724d9e P cd3c04ebb6c84b15a1b3adc748737094 -> Peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Couldn't send request to peer 8fc5d8222c164a679e7a99966da1eaaf. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:13:23.107507 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:23.253125 581 raft_consensus.cc:482] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 343a85ba3b514663af5320cd14a9e144)
I20240501 14:13:23.253312 581 raft_consensus.cc:504] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "a0e901103604414c829b40edf9f24c28" member_type: VOTER last_known_addr { host: "127.16.36.135" port: 42127 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:23.253964 581 leader_election.cc:288] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127)
W20240501 14:13:23.256157 29073 leader_election.cc:334] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111)
W20240501 14:13:23.256510 29073 leader_election.cc:334] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
I20240501 14:13:23.256682 29073 leader_election.cc:302] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 343a85ba3b514663af5320cd14a9e144, a0e901103604414c829b40edf9f24c28
I20240501 14:13:23.257071 581 raft_consensus.cc:2726] T 6aaf52ee71b44511b7b8909711ab0a42 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:13:23.360657 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:23.399578 29073 consensus_peers.cc:485] T f4402de0b48041c383568feb9365147e P 3a3972b170c54344a4ba61f176f837ba -> Peer 26acba62745c48e3ba8180748d55b211 (127.16.36.133:40601): Couldn't send request to peer 26acba62745c48e3ba8180748d55b211. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.133:40601: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 14:13:23.435967 28589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 14:13:23.441612 16530 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 28794: Timed out: CheckLeaks RPC to 127.16.36.130:37325 timed out after 1.000s (SENT)
I20240501 14:13:23.441965 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 28794
I20240501 14:13:23.473071 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 28925
I20240501 14:13:23.483625 29044 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:13:23.592025 28589 catalog_manager.cc:4573] TS 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet efe7fed8e5d44e8a80ffbdc27619a74d: Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
I20240501 14:13:23.674949 581 raft_consensus.cc:482] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8fc5d8222c164a679e7a99966da1eaaf)
I20240501 14:13:23.675323 581 raft_consensus.cc:504] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "8fc5d8222c164a679e7a99966da1eaaf" member_type: VOTER last_known_addr { host: "127.16.36.134" port: 44241 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:23.676550 581 leader_election.cc:288] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
W20240501 14:13:23.678403 29072 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8fc5d8222c164a679e7a99966da1eaaf (127.16.36.134:44241): Network error: Client connection negotiation failed: client connection to 127.16.36.134:44241: connect: Connection refused (error 111)
W20240501 14:13:23.789073 29073 consensus_peers.cc:462] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:23.849164 29071 connection.cc:535] client connection to 127.16.36.130:37325 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:13:23.850678 29071 leader_election.cc:334] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Network error: Client connection negotiation failed: client connection to 127.16.36.130:37325: connect: Connection refused (error 111)
I20240501 14:13:23.850855 29071 leader_election.cc:302] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 8fc5d8222c164a679e7a99966da1eaaf, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:13:23.851207 581 raft_consensus.cc:2726] T 63b7f2a8d8a84faf86c4de16b3cc6663 P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:13:23.873857 581 raft_consensus.cc:482] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 343a85ba3b514663af5320cd14a9e144)
I20240501 14:13:23.874136 581 raft_consensus.cc:504] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "343a85ba3b514663af5320cd14a9e144" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 41051 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } } peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } }
I20240501 14:13:23.874998 581 leader_election.cc:288] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051), cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325)
W20240501 14:13:23.877027 29073 leader_election.cc:334] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 343a85ba3b514663af5320cd14a9e144 (127.16.36.129:41051): Network error: Client connection negotiation failed: client connection to 127.16.36.129:41051: connect: Connection refused (error 111)
W20240501 14:13:23.877312 29071 leader_election.cc:334] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd3c04ebb6c84b15a1b3adc748737094 (127.16.36.130:37325): Network error: Client connection negotiation failed: client connection to 127.16.36.130:37325: connect: Connection refused (error 111)
I20240501 14:13:23.877439 29071 leader_election.cc:302] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3972b170c54344a4ba61f176f837ba; no voters: 343a85ba3b514663af5320cd14a9e144, cd3c04ebb6c84b15a1b3adc748737094
I20240501 14:13:23.877722 581 raft_consensus.cc:2726] T 9b473e10a8ad4bd58bb44aabd9656e08 P 3a3972b170c54344a4ba61f176f837ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
LLVM ERROR: IO failure on output stream: Broken pipe
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
/tmp/dist-test-task1g_oHS/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcdfce)[0x561a90fdffce]
/tmp/dist-test-task1g_oHS/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc27d)[0x561a90fde27d]
/tmp/dist-test-task1g_oHS/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xcc3d3)[0x561a90fde3d3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fccd855c980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fccd763afb7]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fccd763c921]
/tmp/dist-test-task1g_oHS/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa575c)[0x561a90fb775c]
/tmp/dist-test-task1g_oHS/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xa58e0)[0x561a90fb78e0]
/tmp/dist-test-task1g_oHS/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0xbcc73)[0x561a90fcec73]
/lib/x86_64-linux-gnu/libc.so.6(+0x43161)[0x7fccd763f161]
/lib/x86_64-linux-gnu/libc.so.6(+0x4325a)[0x7fccd763f25a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xee)[0x7fccd761dbfe]
/tmp/dist-test-task1g_oHS/thirdparty/installed/uninstrumented/bin/llvm-symbolizer(+0x8f0ba)[0x561a90fa10ba]
W20240501 14:13:23.964095 28942 consensus_peers.cc:462] T efe7fed8e5d44e8a80ffbdc27619a74d P 7e7bc120bd74479ab378f779778fd1e8 -> Peer a0e901103604414c829b40edf9f24c28 (127.16.36.135:42127): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.135:42127: connect: Connection refused (error 111)
W20240501 14:13:23.964243 28942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.16.36.135:42127, user_credentials={real_user=slave}} blocked reactor thread for 479671us
I20240501 14:13:23.972502 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 28925
W20240501 14:13:24.031533 29072 connection.cc:535] client connection to 127.16.36.131:41159 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:13:24.031736 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 29056
W20240501 14:13:24.033015 29072 consensus_peers.cc:485] T 6b6a54ac32694640bffdf1c7f3b035cc P 3a3972b170c54344a4ba61f176f837ba -> Peer 7e7bc120bd74479ab378f779778fd1e8 (127.16.36.131:41159): Couldn't send request to peer 7e7bc120bd74479ab378f779778fd1e8. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.131:41159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:13:24.039520 29178 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:24.468401 782 raft_consensus.cc:482] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cd3c04ebb6c84b15a1b3adc748737094)
I20240501 14:13:24.468752 782 raft_consensus.cc:504] T 135da52054584a27959efdd9126cfcef P 3a3972b170c54344a4ba61f176f837ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3c04ebb6c84b15a1b3adc748737094" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 37325 } } peers { permanent_uuid: "7e7bc120bd74479ab378f779778fd1e8" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 41159 } } peers { permanent_uuid: "3a3972b170c54344a4ba61f176f837ba" member_type: VOTER last_known_addr { host: "127.16.36.132" port: 42827 } }
I20240501 14:13:24.469377 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 29056
I20240501 14:13:24.532912 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 28573
I20240501 14:13:24.542892 28634 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:24.973664 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 28573
2024-05-01T14:13:25Z chronyd exiting
I20240501 14:13:25.055879 16530 test_util.cc:175] -----------------------------------------------
I20240501 14:13:25.055987 16530 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714571953216231-16530-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (681636 ms)
[----------] 1 test from ConcurrentRebalancersTest (681636 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240501 14:13:25.058370 16530 test_util.cc:268] Using random seed: 385658662
I20240501 14:13:25.060236 16530 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:13:25.060307 16530 ts_itest-base.cc:113] --------------
I20240501 14:13:25.060349 16530 ts_itest-base.cc:114] 4 tablet servers
I20240501 14:13:25.060391 16530 ts_itest-base.cc:115] 1 replicas per TS
I20240501 14:13:25.060432 16530 ts_itest-base.cc:116] --------------
2024-05-01T14:13:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:13:25Z Disabled control of system clock
I20240501 14:13:25.125339 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:43297
--webserver_interface=127.16.36.190
--webserver_port=0
--builtin_ntp_servers=127.16.36.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.190:43297
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240501 14:13:25.504597 793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:25.504940 793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:25.505074 793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:25.512996 793 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:13:25.513135 793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:13:25.513213 793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:25.513301 793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:13:25.513367 793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:13:25.523470 793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.190:43297
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:43297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:25.524911 793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:25.526818 793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:25.538741 801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:25.539314 799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:25.539458 798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:25.540025 793 server_base.cc:1034] running on GCE node
I20240501 14:13:25.540866 793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:25.542353 793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:25.543553 793 hybrid_clock.cc:648] HybridClock initialized: now 1714572805543485 us; error 66 us; skew 500 ppm
I20240501 14:13:25.543920 793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:25.549765 793 webserver.cc:460] Webserver started at http://127.16.36.190:43079/ using document root <none> and password file <none>
I20240501 14:13:25.550743 793 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:25.550890 793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:25.551293 793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:25.554481 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d4cf24e21e0546beb7c379f1a0034d52"
format_stamp: "Formatted at 2024-05-01 14:13:25 on dist-test-slave-9q9t"
I20240501 14:13:25.555521 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d4cf24e21e0546beb7c379f1a0034d52"
format_stamp: "Formatted at 2024-05-01 14:13:25 on dist-test-slave-9q9t"
I20240501 14:13:25.561973 793 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240501 14:13:25.565990 807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:25.567587 793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:25.567795 793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d4cf24e21e0546beb7c379f1a0034d52"
format_stamp: "Formatted at 2024-05-01 14:13:25 on dist-test-slave-9q9t"
I20240501 14:13:25.568038 793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:25.596639 793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:25.597577 793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:25.597887 793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:25.617033 793 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.190:43297
I20240501 14:13:25.617051 858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.190:43297 every 8 connection(s)
I20240501 14:13:25.618453 793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:13:25.622434 859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:25.624859 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 793
I20240501 14:13:25.625069 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:13:25.635354 859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52: Bootstrap starting.
I20240501 14:13:25.639330 859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:25.640411 859 log.cc:824] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52: Log is configured to *not* fsync() on all Append() calls
I20240501 14:13:25.643007 859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52: No bootstrap required, opened a new log
I20240501 14:13:25.649240 859 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf24e21e0546beb7c379f1a0034d52" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43297 } }
I20240501 14:13:25.649652 859 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:25.649756 859 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cf24e21e0546beb7c379f1a0034d52, State: Initialized, Role: FOLLOWER
I20240501 14:13:25.650358 859 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf24e21e0546beb7c379f1a0034d52" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43297 } }
I20240501 14:13:25.650568 859 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:25.650712 859 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:25.650864 859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:25.652964 859 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf24e21e0546beb7c379f1a0034d52" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43297 } }
I20240501 14:13:25.653450 859 leader_election.cc:302] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4cf24e21e0546beb7c379f1a0034d52; no voters:
I20240501 14:13:25.654035 859 leader_election.cc:288] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:25.654181 864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:25.654932 864 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [term 1 LEADER]: Becoming Leader. State: Replica: d4cf24e21e0546beb7c379f1a0034d52, State: Running, Role: LEADER
I20240501 14:13:25.655500 864 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf24e21e0546beb7c379f1a0034d52" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43297 } }
I20240501 14:13:25.656196 859 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:13:25.658563 866 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4cf24e21e0546beb7c379f1a0034d52. Latest consensus state: current_term: 1 leader_uuid: "d4cf24e21e0546beb7c379f1a0034d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf24e21e0546beb7c379f1a0034d52" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43297 } } }
I20240501 14:13:25.658530 865 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4cf24e21e0546beb7c379f1a0034d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf24e21e0546beb7c379f1a0034d52" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43297 } } }
I20240501 14:13:25.659070 865 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [sys.catalog]: This master's current role is: LEADER
I20240501 14:13:25.659617 866 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52 [sys.catalog]: This master's current role is: LEADER
I20240501 14:13:25.661005 871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:13:25.666087 871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:13:25.674429 871 catalog_manager.cc:1347] Generated new cluster ID: f5a3e553256743c290cb0762baf6a4e2
I20240501 14:13:25.674551 871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:13:25.687541 871 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:13:25.688834 871 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:13:25.696790 871 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d4cf24e21e0546beb7c379f1a0034d52: Generated new TSK 0
I20240501 14:13:25.697649 871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:13:25.702735 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.129:0
--local_ip_for_outbound_sockets=127.16.36.129
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--builtin_ntp_servers=127.16.36.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:13:26.064497 883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:26.064807 883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:26.064925 883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:26.072618 883 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:13:26.072752 883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:26.072913 883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.129
I20240501 14:13:26.082260 883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.129
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:26.083642 883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:26.085403 883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:26.098055 888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:26.098659 889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:26.098809 891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:26.099304 883 server_base.cc:1034] running on GCE node
I20240501 14:13:26.099906 883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:26.100863 883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:26.102072 883 hybrid_clock.cc:648] HybridClock initialized: now 1714572806102008 us; error 56 us; skew 500 ppm
I20240501 14:13:26.102417 883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:26.104868 883 webserver.cc:460] Webserver started at http://127.16.36.129:45997/ using document root <none> and password file <none>
I20240501 14:13:26.105648 883 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:26.105787 883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:26.106112 883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:26.108832 883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9cd259d813db47c89bcb8b6f55a87fc7"
format_stamp: "Formatted at 2024-05-01 14:13:26 on dist-test-slave-9q9t"
I20240501 14:13:26.109616 883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9cd259d813db47c89bcb8b6f55a87fc7"
format_stamp: "Formatted at 2024-05-01 14:13:26 on dist-test-slave-9q9t"
I20240501 14:13:26.114964 883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20240501 14:13:26.118376 897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:26.119807 883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:26.119971 883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9cd259d813db47c89bcb8b6f55a87fc7"
format_stamp: "Formatted at 2024-05-01 14:13:26 on dist-test-slave-9q9t"
I20240501 14:13:26.120179 883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:26.158268 883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:26.159262 883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:26.159567 883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:26.160734 883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:26.162473 883 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:26.162577 883 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:26.162717 883 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:26.162799 883 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:26.196529 883 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.129:38425
I20240501 14:13:26.196650 1009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.129:38425 every 8 connection(s)
I20240501 14:13:26.197954 883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:13:26.200919 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 883
I20240501 14:13:26.201153 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:13:26.204843 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.130:0
--local_ip_for_outbound_sockets=127.16.36.130
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--builtin_ntp_servers=127.16.36.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:13:26.212157 1010 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43297
I20240501 14:13:26.212551 1010 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:26.213572 1010 heartbeater.cc:511] Master 127.16.36.190:43297 requested a full tablet report, sending...
I20240501 14:13:26.215876 824 ts_manager.cc:194] Registered new tserver with Master: 9cd259d813db47c89bcb8b6f55a87fc7 (127.16.36.129:38425)
I20240501 14:13:26.217411 824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.129:43837
W20240501 14:13:26.570498 1014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:26.570899 1014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:26.571049 1014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:26.579039 1014 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:13:26.579185 1014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:26.579355 1014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.130
I20240501 14:13:26.588838 1014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.130
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:26.590251 1014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:26.592190 1014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:26.604518 1019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:26.605111 1020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:26.605717 1022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:26.606040 1014 server_base.cc:1034] running on GCE node
I20240501 14:13:26.606715 1014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:26.607705 1014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:26.608927 1014 hybrid_clock.cc:648] HybridClock initialized: now 1714572806608875 us; error 49 us; skew 500 ppm
I20240501 14:13:26.609254 1014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:26.611603 1014 webserver.cc:460] Webserver started at http://127.16.36.130:41137/ using document root <none> and password file <none>
I20240501 14:13:26.612363 1014 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:26.612490 1014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:26.612800 1014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:26.615594 1014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "51efb78072074a22939c0b33c86b64f5"
format_stamp: "Formatted at 2024-05-01 14:13:26 on dist-test-slave-9q9t"
I20240501 14:13:26.616513 1014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "51efb78072074a22939c0b33c86b64f5"
format_stamp: "Formatted at 2024-05-01 14:13:26 on dist-test-slave-9q9t"
I20240501 14:13:26.623329 1014 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240501 14:13:26.627914 1028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:26.629457 1014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:13:26.629633 1014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "51efb78072074a22939c0b33c86b64f5"
format_stamp: "Formatted at 2024-05-01 14:13:26 on dist-test-slave-9q9t"
I20240501 14:13:26.629837 1014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:26.651163 1014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:26.652092 1014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:26.652370 1014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:26.653376 1014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:26.654973 1014 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:26.655074 1014 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:26.655182 1014 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:26.655252 1014 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:26.687024 1014 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.130:38333
I20240501 14:13:26.687076 1140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.130:38333 every 8 connection(s)
I20240501 14:13:26.688373 1014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:13:26.695259 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 1014
I20240501 14:13:26.695426 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:13:26.699041 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.131:0
--local_ip_for_outbound_sockets=127.16.36.131
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--builtin_ntp_servers=127.16.36.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:13:26.701892 1141 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43297
I20240501 14:13:26.702203 1141 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:26.702998 1141 heartbeater.cc:511] Master 127.16.36.190:43297 requested a full tablet report, sending...
I20240501 14:13:26.704457 824 ts_manager.cc:194] Registered new tserver with Master: 51efb78072074a22939c0b33c86b64f5 (127.16.36.130:38333)
I20240501 14:13:26.705613 824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.130:41579
W20240501 14:13:27.060493 1144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:27.060819 1144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:27.060930 1144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:27.068684 1144 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:13:27.068817 1144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:27.068943 1144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.131
I20240501 14:13:27.078738 1144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.131
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:27.080072 1144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:27.081815 1144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:27.093968 1149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:27.094583 1152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:27.094940 1150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:27.095239 1144 server_base.cc:1034] running on GCE node
I20240501 14:13:27.095873 1144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:27.096836 1144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:27.098038 1144 hybrid_clock.cc:648] HybridClock initialized: now 1714572807097978 us; error 58 us; skew 500 ppm
I20240501 14:13:27.098354 1144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:27.100705 1144 webserver.cc:460] Webserver started at http://127.16.36.131:39555/ using document root <none> and password file <none>
I20240501 14:13:27.101449 1144 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:27.101573 1144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:27.101888 1144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:27.104565 1144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0b5de4ca3a414e21ab0a898633a1bffd"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:27.105376 1144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0b5de4ca3a414e21ab0a898633a1bffd"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:27.110864 1144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 14:13:27.114480 1158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:27.115957 1144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:27.116130 1144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0b5de4ca3a414e21ab0a898633a1bffd"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:27.116341 1144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:27.140278 1144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:27.141191 1144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:27.141469 1144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:27.142515 1144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:27.144129 1144 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:27.144234 1144 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:27.144342 1144 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:27.144418 1144 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:27.177884 1144 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.131:45217
I20240501 14:13:27.177954 1270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.131:45217 every 8 connection(s)
I20240501 14:13:27.179327 1144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:13:27.188984 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 1144
I20240501 14:13:27.189189 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:13:27.193107 1271 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43297
I20240501 14:13:27.193187 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:0
--local_ip_for_outbound_sockets=127.16.36.132
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--builtin_ntp_servers=127.16.36.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:13:27.193464 1271 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:27.194245 1271 heartbeater.cc:511] Master 127.16.36.190:43297 requested a full tablet report, sending...
I20240501 14:13:27.195760 824 ts_manager.cc:194] Registered new tserver with Master: 0b5de4ca3a414e21ab0a898633a1bffd (127.16.36.131:45217)
I20240501 14:13:27.196602 824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.131:59139
I20240501 14:13:27.221033 1010 heartbeater.cc:503] Master 127.16.36.190:43297 was elected leader, sending a full tablet report...
W20240501 14:13:27.558873 1274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:27.559198 1274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:27.559295 1274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:27.567324 1274 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:13:27.567471 1274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:27.567592 1274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 14:13:27.577158 1274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:27.578567 1274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:27.580444 1274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:27.592965 1279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:27.593466 1282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:27.594002 1280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:27.594586 1274 server_base.cc:1034] running on GCE node
I20240501 14:13:27.595247 1274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:27.596204 1274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:27.597414 1274 hybrid_clock.cc:648] HybridClock initialized: now 1714572807597354 us; error 60 us; skew 500 ppm
I20240501 14:13:27.597750 1274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:27.600198 1274 webserver.cc:460] Webserver started at http://127.16.36.132:34443/ using document root <none> and password file <none>
I20240501 14:13:27.600991 1274 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:27.601122 1274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:27.601434 1274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:27.604331 1274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "922699286c8742c792604bd5db1378fb"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:27.605152 1274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "922699286c8742c792604bd5db1378fb"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:27.610464 1274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240501 14:13:27.613988 1288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:27.615378 1274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:27.615535 1274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "922699286c8742c792604bd5db1378fb"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:27.615759 1274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:27.631759 1274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:27.632643 1274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:27.632930 1274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:27.633989 1274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:27.635609 1274 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:27.635716 1274 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:27.635828 1274 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:27.635900 1274 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:27.668896 1274 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:33275
I20240501 14:13:27.668967 1400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:33275 every 8 connection(s)
I20240501 14:13:27.670251 1274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:13:27.674814 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 1274
I20240501 14:13:27.674995 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:13:27.682929 1401 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43297
I20240501 14:13:27.683244 1401 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:27.684029 1401 heartbeater.cc:511] Master 127.16.36.190:43297 requested a full tablet report, sending...
I20240501 14:13:27.685396 824 ts_manager.cc:194] Registered new tserver with Master: 922699286c8742c792604bd5db1378fb (127.16.36.132:33275)
I20240501 14:13:27.686239 824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:42169
I20240501 14:13:27.691006 16530 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:13:27.708250 1141 heartbeater.cc:503] Master 127.16.36.190:43297 was elected leader, sending a full tablet report...
I20240501 14:13:27.707062 824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37674:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:13:27.736635 1076 tablet_service.cc:1467] Processing CreateTablet for tablet e5904c475ab54bd3b358115808705b8d (DEFAULT_TABLE table=TestTable [id=b14e7c4618ba44aca33849c389d80f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:27.738091 1076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5904c475ab54bd3b358115808705b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:27.749042 1415 tablet_bootstrap.cc:492] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5: Bootstrap starting.
I20240501 14:13:27.752569 1415 tablet_bootstrap.cc:654] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:27.753657 1415 log.cc:824] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5: Log is configured to *not* fsync() on all Append() calls
I20240501 14:13:27.756263 1415 tablet_bootstrap.cc:492] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5: No bootstrap required, opened a new log
I20240501 14:13:27.756529 1415 ts_tablet_manager.cc:1367] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 14:13:27.763012 1415 raft_consensus.cc:348] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:27.763379 1415 raft_consensus.cc:374] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:27.763481 1415 raft_consensus.cc:729] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Initialized, Role: FOLLOWER
I20240501 14:13:27.764111 1415 consensus_queue.cc:260] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:27.764328 1415 raft_consensus.cc:388] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:27.764463 1415 raft_consensus.cc:482] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:27.764600 1415 raft_consensus.cc:3037] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:27.766870 1415 raft_consensus.cc:504] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:27.767413 1415 leader_election.cc:302] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51efb78072074a22939c0b33c86b64f5; no voters:
I20240501 14:13:27.768051 1415 leader_election.cc:288] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:27.768220 1417 raft_consensus.cc:2781] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:27.768929 1417 raft_consensus.cc:686] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [term 1 LEADER]: Becoming Leader. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Running, Role: LEADER
I20240501 14:13:27.769608 1417 consensus_queue.cc:237] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:27.769840 1415 ts_tablet_manager.cc:1398] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5: Time spent starting tablet: real 0.013s user 0.000s sys 0.014s
I20240501 14:13:27.774850 824 catalog_manager.cc:5484] T e5904c475ab54bd3b358115808705b8d P 51efb78072074a22939c0b33c86b64f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51efb78072074a22939c0b33c86b64f5 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "51efb78072074a22939c0b33c86b64f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } } }
I20240501 14:13:27.795982 16530 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 14:13:27.798151 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9cd259d813db47c89bcb8b6f55a87fc7 to finish bootstrapping
I20240501 14:13:27.807677 16530 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 51efb78072074a22939c0b33c86b64f5 to finish bootstrapping
I20240501 14:13:27.815537 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0b5de4ca3a414e21ab0a898633a1bffd to finish bootstrapping
I20240501 14:13:27.825229 16530 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 922699286c8742c792604bd5db1378fb to finish bootstrapping
I20240501 14:13:27.834529 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 1144
I20240501 14:13:27.835595 1266 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:27.920928 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 1144
I20240501 14:13:27.939800 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 1274
I20240501 14:13:27.940935 1396 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:28.023542 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 1274
I20240501 14:13:31.793632 824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37674:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:13:31.813020 944 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4504b3feee486086d3cbc5701ed520 (DEFAULT_TABLE table=rebalance_test_table_0 [id=32f105b7ca0d470685e769beb09cb1c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:31.813047 943 tablet_service.cc:1467] Processing CreateTablet for tablet 74d6e9743fe64685a3c485c898e933fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=32f105b7ca0d470685e769beb09cb1c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:31.812923 945 tablet_service.cc:1467] Processing CreateTablet for tablet 52531661b6e145c99569da0409a82a56 (DEFAULT_TABLE table=rebalance_test_table_0 [id=32f105b7ca0d470685e769beb09cb1c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:31.814522 944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4504b3feee486086d3cbc5701ed520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:31.815137 943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6e9743fe64685a3c485c898e933fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:31.815452 945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52531661b6e145c99569da0409a82a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:31.828140 1434 tablet_bootstrap.cc:492] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7: Bootstrap starting.
I20240501 14:13:31.831432 1434 tablet_bootstrap.cc:654] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:31.832834 1434 log.cc:824] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:13:31.835793 1434 tablet_bootstrap.cc:492] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7: No bootstrap required, opened a new log
I20240501 14:13:31.836092 1434 ts_tablet_manager.cc:1367] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:13:31.842468 1434 raft_consensus.cc:348] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.842917 1434 raft_consensus.cc:374] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:31.843044 1434 raft_consensus.cc:729] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Initialized, Role: FOLLOWER
I20240501 14:13:31.843760 1434 consensus_queue.cc:260] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.843981 1434 raft_consensus.cc:388] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:31.844138 1434 raft_consensus.cc:482] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:31.844318 1434 raft_consensus.cc:3037] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:31.846598 1434 raft_consensus.cc:504] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.847146 1434 leader_election.cc:302] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd259d813db47c89bcb8b6f55a87fc7; no voters:
I20240501 14:13:31.847757 1434 leader_election.cc:288] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:31.847937 1436 raft_consensus.cc:2781] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:31.848594 1436 raft_consensus.cc:686] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Running, Role: LEADER
I20240501 14:13:31.849180 1436 consensus_queue.cc:237] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.849560 1434 ts_tablet_manager.cc:1398] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20240501 14:13:31.849977 1434 tablet_bootstrap.cc:492] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7: Bootstrap starting.
I20240501 14:13:31.852056 1434 tablet_bootstrap.cc:654] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:31.854175 1434 tablet_bootstrap.cc:492] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7: No bootstrap required, opened a new log
I20240501 14:13:31.854300 1434 ts_tablet_manager.cc:1367] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:13:31.854264 824 catalog_manager.cc:5484] T 52531661b6e145c99569da0409a82a56 P 9cd259d813db47c89bcb8b6f55a87fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd259d813db47c89bcb8b6f55a87fc7 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } } }
I20240501 14:13:31.855216 1434 raft_consensus.cc:348] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.855383 1434 raft_consensus.cc:374] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:31.855479 1434 raft_consensus.cc:729] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Initialized, Role: FOLLOWER
I20240501 14:13:31.855721 1434 consensus_queue.cc:260] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.855898 1434 raft_consensus.cc:388] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:31.855985 1434 raft_consensus.cc:482] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:31.856138 1434 raft_consensus.cc:3037] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:31.858588 1434 raft_consensus.cc:504] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.858882 1434 leader_election.cc:302] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd259d813db47c89bcb8b6f55a87fc7; no voters:
I20240501 14:13:31.859092 1434 leader_election.cc:288] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:31.859179 1436 raft_consensus.cc:2781] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:31.859580 1434 ts_tablet_manager.cc:1398] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:13:31.859483 1436 raft_consensus.cc:686] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Running, Role: LEADER
I20240501 14:13:31.859860 1434 tablet_bootstrap.cc:492] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7: Bootstrap starting.
I20240501 14:13:31.859863 1436 consensus_queue.cc:237] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.861805 1434 tablet_bootstrap.cc:654] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:31.862452 824 catalog_manager.cc:5484] T 5e4504b3feee486086d3cbc5701ed520 P 9cd259d813db47c89bcb8b6f55a87fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd259d813db47c89bcb8b6f55a87fc7 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } } }
I20240501 14:13:31.864545 1434 tablet_bootstrap.cc:492] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7: No bootstrap required, opened a new log
I20240501 14:13:31.864718 1434 ts_tablet_manager.cc:1367] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:13:31.865428 1434 raft_consensus.cc:348] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.865634 1434 raft_consensus.cc:374] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:31.865731 1434 raft_consensus.cc:729] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Initialized, Role: FOLLOWER
I20240501 14:13:31.865976 1434 consensus_queue.cc:260] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.866122 1434 raft_consensus.cc:388] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:31.866191 1434 raft_consensus.cc:482] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:31.866256 1434 raft_consensus.cc:3037] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:31.868067 1434 raft_consensus.cc:504] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.868255 1434 leader_election.cc:302] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd259d813db47c89bcb8b6f55a87fc7; no voters:
I20240501 14:13:31.868434 1434 leader_election.cc:288] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:31.868527 1436 raft_consensus.cc:2781] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:31.868691 1436 raft_consensus.cc:686] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Running, Role: LEADER
I20240501 14:13:31.868896 1434 ts_tablet_manager.cc:1398] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:13:31.868953 1436 consensus_queue.cc:237] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:31.871034 824 catalog_manager.cc:5484] T 74d6e9743fe64685a3c485c898e933fa P 9cd259d813db47c89bcb8b6f55a87fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd259d813db47c89bcb8b6f55a87fc7 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } } }
I20240501 14:13:32.438614 824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37674:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:13:32.447788 1076 tablet_service.cc:1467] Processing CreateTablet for tablet 68391462ac6d4a20bdf52c3d838cc0c7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c5b55a66755d4eafa66fe4e613cbde72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:32.447837 1074 tablet_service.cc:1467] Processing CreateTablet for tablet 81d570b279fa4b9baaa6fb4d6fb9578c (DEFAULT_TABLE table=rebalance_test_table_1 [id=c5b55a66755d4eafa66fe4e613cbde72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:32.447748 1075 tablet_service.cc:1467] Processing CreateTablet for tablet 45303dac715e4c83b9a24b7bc2db9236 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c5b55a66755d4eafa66fe4e613cbde72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:32.448318 1076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68391462ac6d4a20bdf52c3d838cc0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:32.448632 1074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d570b279fa4b9baaa6fb4d6fb9578c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:32.448901 1075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45303dac715e4c83b9a24b7bc2db9236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:32.455186 1470 tablet_bootstrap.cc:492] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5: Bootstrap starting.
I20240501 14:13:32.456990 1470 tablet_bootstrap.cc:654] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:32.459070 1470 tablet_bootstrap.cc:492] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5: No bootstrap required, opened a new log
I20240501 14:13:32.459233 1470 ts_tablet_manager.cc:1367] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:13:32.460049 1470 raft_consensus.cc:348] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.460202 1470 raft_consensus.cc:374] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:32.460294 1470 raft_consensus.cc:729] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Initialized, Role: FOLLOWER
I20240501 14:13:32.460497 1470 consensus_queue.cc:260] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.460690 1470 raft_consensus.cc:388] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:32.460762 1470 raft_consensus.cc:482] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:32.460886 1470 raft_consensus.cc:3037] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:32.463172 1470 raft_consensus.cc:504] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.463400 1470 leader_election.cc:302] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51efb78072074a22939c0b33c86b64f5; no voters:
I20240501 14:13:32.463845 1470 leader_election.cc:288] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:32.464040 1472 raft_consensus.cc:2781] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:32.464663 1472 raft_consensus.cc:686] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [term 1 LEADER]: Becoming Leader. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Running, Role: LEADER
I20240501 14:13:32.464664 1470 ts_tablet_manager.cc:1398] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:13:32.465065 1470 tablet_bootstrap.cc:492] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5: Bootstrap starting.
I20240501 14:13:32.465030 1472 consensus_queue.cc:237] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.466732 1470 tablet_bootstrap.cc:654] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:32.468304 823 catalog_manager.cc:5484] T 68391462ac6d4a20bdf52c3d838cc0c7 P 51efb78072074a22939c0b33c86b64f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51efb78072074a22939c0b33c86b64f5 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "51efb78072074a22939c0b33c86b64f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } } }
I20240501 14:13:32.470714 1470 tablet_bootstrap.cc:492] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5: No bootstrap required, opened a new log
I20240501 14:13:32.470947 1470 ts_tablet_manager.cc:1367] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20240501 14:13:32.471967 1470 raft_consensus.cc:348] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.472211 1470 raft_consensus.cc:374] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:32.472304 1470 raft_consensus.cc:729] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Initialized, Role: FOLLOWER
I20240501 14:13:32.472674 1470 consensus_queue.cc:260] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.472875 1470 raft_consensus.cc:388] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:32.473042 1470 raft_consensus.cc:482] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:32.473171 1470 raft_consensus.cc:3037] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:32.476056 1470 raft_consensus.cc:504] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.476284 1470 leader_election.cc:302] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51efb78072074a22939c0b33c86b64f5; no voters:
I20240501 14:13:32.476496 1470 leader_election.cc:288] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:32.476558 1472 raft_consensus.cc:2781] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:32.476751 1472 raft_consensus.cc:686] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [term 1 LEADER]: Becoming Leader. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Running, Role: LEADER
I20240501 14:13:32.476999 1472 consensus_queue.cc:237] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.477056 1470 ts_tablet_manager.cc:1398] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:13:32.477397 1470 tablet_bootstrap.cc:492] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5: Bootstrap starting.
I20240501 14:13:32.479490 1470 tablet_bootstrap.cc:654] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:32.479286 823 catalog_manager.cc:5484] T 45303dac715e4c83b9a24b7bc2db9236 P 51efb78072074a22939c0b33c86b64f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51efb78072074a22939c0b33c86b64f5 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "51efb78072074a22939c0b33c86b64f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } } }
I20240501 14:13:32.487679 1470 tablet_bootstrap.cc:492] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5: No bootstrap required, opened a new log
I20240501 14:13:32.487818 1470 ts_tablet_manager.cc:1367] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 14:13:32.488408 1470 raft_consensus.cc:348] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.488559 1470 raft_consensus.cc:374] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:32.488652 1470 raft_consensus.cc:729] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Initialized, Role: FOLLOWER
I20240501 14:13:32.488917 1470 consensus_queue.cc:260] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.489146 1470 raft_consensus.cc:388] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:32.489225 1470 raft_consensus.cc:482] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:32.489333 1470 raft_consensus.cc:3037] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:32.491293 1470 raft_consensus.cc:504] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.491556 1470 leader_election.cc:302] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51efb78072074a22939c0b33c86b64f5; no voters:
I20240501 14:13:32.491735 1470 leader_election.cc:288] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:32.491890 1472 raft_consensus.cc:2781] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:32.492148 1472 raft_consensus.cc:686] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [term 1 LEADER]: Becoming Leader. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Running, Role: LEADER
I20240501 14:13:32.492174 1470 ts_tablet_manager.cc:1398] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:13:32.492410 1472 consensus_queue.cc:237] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:32.494683 823 catalog_manager.cc:5484] T 81d570b279fa4b9baaa6fb4d6fb9578c P 51efb78072074a22939c0b33c86b64f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51efb78072074a22939c0b33c86b64f5 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "51efb78072074a22939c0b33c86b64f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } } }
I20240501 14:13:33.030566 823 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37674:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:13:33.039556 945 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9f9d396afa448bb979788eac3f7f86 (DEFAULT_TABLE table=rebalance_test_table_2 [id=49e867fb71ec4cf4b811c10bb105adf7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:33.039749 943 tablet_service.cc:1467] Processing CreateTablet for tablet 30181ff3c42b4d5c8315daa414f170bf (DEFAULT_TABLE table=rebalance_test_table_2 [id=49e867fb71ec4cf4b811c10bb105adf7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:33.040105 945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9f9d396afa448bb979788eac3f7f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:33.040326 943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30181ff3c42b4d5c8315daa414f170bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:33.040274 944 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb30806c2284c4594cbe4d62c49e001 (DEFAULT_TABLE table=rebalance_test_table_2 [id=49e867fb71ec4cf4b811c10bb105adf7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:33.040603 944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb30806c2284c4594cbe4d62c49e001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:33.047689 1505 tablet_bootstrap.cc:492] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7: Bootstrap starting.
I20240501 14:13:33.048982 1505 tablet_bootstrap.cc:654] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:33.050844 1505 tablet_bootstrap.cc:492] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7: No bootstrap required, opened a new log
I20240501 14:13:33.051007 1505 ts_tablet_manager.cc:1367] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:33.051666 1505 raft_consensus.cc:348] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.051815 1505 raft_consensus.cc:374] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:33.051913 1505 raft_consensus.cc:729] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Initialized, Role: FOLLOWER
I20240501 14:13:33.052160 1505 consensus_queue.cc:260] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.052318 1505 raft_consensus.cc:388] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:33.052394 1505 raft_consensus.cc:482] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:33.052500 1505 raft_consensus.cc:3037] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:33.054333 1505 raft_consensus.cc:504] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.054606 1505 leader_election.cc:302] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd259d813db47c89bcb8b6f55a87fc7; no voters:
I20240501 14:13:33.055104 1505 leader_election.cc:288] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:33.055261 1507 raft_consensus.cc:2781] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:33.055778 1507 raft_consensus.cc:686] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Running, Role: LEADER
I20240501 14:13:33.056166 1507 consensus_queue.cc:237] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.058559 823 catalog_manager.cc:5484] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd259d813db47c89bcb8b6f55a87fc7 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } } }
I20240501 14:13:33.062006 1505 ts_tablet_manager.cc:1398] T 3b9f9d396afa448bb979788eac3f7f86 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s
I20240501 14:13:33.062322 1505 tablet_bootstrap.cc:492] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7: Bootstrap starting.
I20240501 14:13:33.063741 1505 tablet_bootstrap.cc:654] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:33.065527 1505 tablet_bootstrap.cc:492] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7: No bootstrap required, opened a new log
I20240501 14:13:33.065703 1505 ts_tablet_manager.cc:1367] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:13:33.066561 1505 raft_consensus.cc:348] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.066781 1505 raft_consensus.cc:374] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:33.066862 1505 raft_consensus.cc:729] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Initialized, Role: FOLLOWER
I20240501 14:13:33.067095 1505 consensus_queue.cc:260] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.067245 1505 raft_consensus.cc:388] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:33.067338 1505 raft_consensus.cc:482] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:33.067430 1505 raft_consensus.cc:3037] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:33.069232 1505 raft_consensus.cc:504] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.069444 1505 leader_election.cc:302] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd259d813db47c89bcb8b6f55a87fc7; no voters:
I20240501 14:13:33.069631 1505 leader_election.cc:288] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:33.069733 1508 raft_consensus.cc:2781] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:33.070040 1505 ts_tablet_manager.cc:1398] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:13:33.070140 1508 raft_consensus.cc:686] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Running, Role: LEADER
I20240501 14:13:33.070320 1505 tablet_bootstrap.cc:492] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7: Bootstrap starting.
I20240501 14:13:33.070400 1508 consensus_queue.cc:237] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.072010 1505 tablet_bootstrap.cc:654] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:33.073086 823 catalog_manager.cc:5484] T 30181ff3c42b4d5c8315daa414f170bf P 9cd259d813db47c89bcb8b6f55a87fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd259d813db47c89bcb8b6f55a87fc7 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } } }
I20240501 14:13:33.074254 1505 tablet_bootstrap.cc:492] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7: No bootstrap required, opened a new log
I20240501 14:13:33.074373 1505 ts_tablet_manager.cc:1367] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:13:33.075093 1505 raft_consensus.cc:348] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.075227 1505 raft_consensus.cc:374] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:33.075356 1505 raft_consensus.cc:729] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Initialized, Role: FOLLOWER
I20240501 14:13:33.075573 1505 consensus_queue.cc:260] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.075764 1505 raft_consensus.cc:388] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:33.075853 1505 raft_consensus.cc:482] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:33.075975 1505 raft_consensus.cc:3037] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:33.078002 1505 raft_consensus.cc:504] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.078235 1505 leader_election.cc:302] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd259d813db47c89bcb8b6f55a87fc7; no voters:
I20240501 14:13:33.078411 1505 leader_election.cc:288] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:33.078501 1508 raft_consensus.cc:2781] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:33.078692 1508 raft_consensus.cc:686] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd259d813db47c89bcb8b6f55a87fc7, State: Running, Role: LEADER
I20240501 14:13:33.078982 1505 ts_tablet_manager.cc:1398] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 14:13:33.078936 1508 consensus_queue.cc:237] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } }
I20240501 14:13:33.081050 823 catalog_manager.cc:5484] T 5eb30806c2284c4594cbe4d62c49e001 P 9cd259d813db47c89bcb8b6f55a87fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd259d813db47c89bcb8b6f55a87fc7 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd259d813db47c89bcb8b6f55a87fc7" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 38425 } } }
I20240501 14:13:34.535224 824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37674:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:13:34.544487 1074 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6e38956cdd4e1897f70cf7e53d0aec (DEFAULT_TABLE table=rebalance_test_table_3 [id=da1ad7db13034efdbdc6494b24ec7fbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:34.544656 1076 tablet_service.cc:1467] Processing CreateTablet for tablet 8494e8aca92e4620b605d4ee2d0969e9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=da1ad7db13034efdbdc6494b24ec7fbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:34.544697 1075 tablet_service.cc:1467] Processing CreateTablet for tablet 50b04bc92b67435c8d429b70224514ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=da1ad7db13034efdbdc6494b24ec7fbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:13:34.545045 1074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6e38956cdd4e1897f70cf7e53d0aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:34.545259 1076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8494e8aca92e4620b605d4ee2d0969e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:34.545576 1075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b04bc92b67435c8d429b70224514ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:34.551882 1542 tablet_bootstrap.cc:492] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5: Bootstrap starting.
I20240501 14:13:34.553457 1542 tablet_bootstrap.cc:654] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:34.555483 1542 tablet_bootstrap.cc:492] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5: No bootstrap required, opened a new log
I20240501 14:13:34.555637 1542 ts_tablet_manager.cc:1367] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:13:34.556309 1542 raft_consensus.cc:348] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.556463 1542 raft_consensus.cc:374] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:34.556558 1542 raft_consensus.cc:729] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Initialized, Role: FOLLOWER
I20240501 14:13:34.556787 1542 consensus_queue.cc:260] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.556952 1542 raft_consensus.cc:388] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:34.557024 1542 raft_consensus.cc:482] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:34.557142 1542 raft_consensus.cc:3037] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:34.559224 1542 raft_consensus.cc:504] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.559484 1542 leader_election.cc:302] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51efb78072074a22939c0b33c86b64f5; no voters:
I20240501 14:13:34.559871 1542 leader_election.cc:288] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:34.560050 1544 raft_consensus.cc:2781] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:34.560565 1542 ts_tablet_manager.cc:1398] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:13:34.560657 1544 raft_consensus.cc:686] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [term 1 LEADER]: Becoming Leader. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Running, Role: LEADER
I20240501 14:13:34.560926 1542 tablet_bootstrap.cc:492] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5: Bootstrap starting.
I20240501 14:13:34.561000 1544 consensus_queue.cc:237] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.562701 1542 tablet_bootstrap.cc:654] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:34.563800 824 catalog_manager.cc:5484] T 50b04bc92b67435c8d429b70224514ee P 51efb78072074a22939c0b33c86b64f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51efb78072074a22939c0b33c86b64f5 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "51efb78072074a22939c0b33c86b64f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } } }
I20240501 14:13:34.566612 1542 tablet_bootstrap.cc:492] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5: No bootstrap required, opened a new log
I20240501 14:13:34.566819 1542 ts_tablet_manager.cc:1367] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:13:34.567593 1542 raft_consensus.cc:348] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.567773 1542 raft_consensus.cc:374] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:34.567854 1542 raft_consensus.cc:729] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Initialized, Role: FOLLOWER
I20240501 14:13:34.568117 1542 consensus_queue.cc:260] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.568266 1542 raft_consensus.cc:388] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:34.568370 1542 raft_consensus.cc:482] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:34.568466 1542 raft_consensus.cc:3037] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:34.570360 1542 raft_consensus.cc:504] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.570616 1542 leader_election.cc:302] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51efb78072074a22939c0b33c86b64f5; no voters:
I20240501 14:13:34.570917 1542 leader_election.cc:288] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:34.570996 1544 raft_consensus.cc:2781] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:34.571501 1544 raft_consensus.cc:686] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [term 1 LEADER]: Becoming Leader. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Running, Role: LEADER
I20240501 14:13:34.571807 1542 ts_tablet_manager.cc:1398] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:13:34.571758 1544 consensus_queue.cc:237] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.572145 1542 tablet_bootstrap.cc:492] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5: Bootstrap starting.
I20240501 14:13:34.573962 1542 tablet_bootstrap.cc:654] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:34.574455 824 catalog_manager.cc:5484] T 8494e8aca92e4620b605d4ee2d0969e9 P 51efb78072074a22939c0b33c86b64f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51efb78072074a22939c0b33c86b64f5 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "51efb78072074a22939c0b33c86b64f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } } }
I20240501 14:13:34.575856 1542 tablet_bootstrap.cc:492] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5: No bootstrap required, opened a new log
I20240501 14:13:34.576027 1542 ts_tablet_manager.cc:1367] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:13:34.576853 1542 raft_consensus.cc:348] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.577040 1542 raft_consensus.cc:374] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:34.577147 1542 raft_consensus.cc:729] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Initialized, Role: FOLLOWER
I20240501 14:13:34.577389 1542 consensus_queue.cc:260] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.577584 1542 raft_consensus.cc:388] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:34.577729 1542 raft_consensus.cc:482] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:34.577860 1542 raft_consensus.cc:3037] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:34.580557 1542 raft_consensus.cc:504] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.580818 1542 leader_election.cc:302] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51efb78072074a22939c0b33c86b64f5; no voters:
I20240501 14:13:34.581003 1542 leader_election.cc:288] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:34.581144 1544 raft_consensus.cc:2781] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:34.581333 1544 raft_consensus.cc:686] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [term 1 LEADER]: Becoming Leader. State: Replica: 51efb78072074a22939c0b33c86b64f5, State: Running, Role: LEADER
I20240501 14:13:34.581449 1542 ts_tablet_manager.cc:1398] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:13:34.581619 1544 consensus_queue.cc:237] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } }
I20240501 14:13:34.583853 824 catalog_manager.cc:5484] T 6d6e38956cdd4e1897f70cf7e53d0aec P 51efb78072074a22939c0b33c86b64f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51efb78072074a22939c0b33c86b64f5 (127.16.36.130). New cstate: current_term: 1 leader_uuid: "51efb78072074a22939c0b33c86b64f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51efb78072074a22939c0b33c86b64f5" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 38333 } } }
I20240501 14:13:35.115725 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.131:45217
--local_ip_for_outbound_sockets=127.16.36.131
--tserver_master_addrs=127.16.36.190:43297
--webserver_port=39555
--webserver_interface=127.16.36.131
--builtin_ntp_servers=127.16.36.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 14:13:35.477341 1574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:35.477676 1574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:35.477813 1574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:35.485832 1574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:13:35.485996 1574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:35.486186 1574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.131
I20240501 14:13:35.495879 1574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.131:45217
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.131
--webserver_port=39555
--tserver_master_addrs=127.16.36.190:43297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.131
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:35.497329 1574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:35.499194 1574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:35.512056 1580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:35.512616 1579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:35.513089 1582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:35.513828 1574 server_base.cc:1034] running on GCE node
I20240501 14:13:35.514447 1574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:35.515506 1574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:35.516729 1574 hybrid_clock.cc:648] HybridClock initialized: now 1714572815516677 us; error 45 us; skew 500 ppm
I20240501 14:13:35.517089 1574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:35.519609 1574 webserver.cc:460] Webserver started at http://127.16.36.131:39555/ using document root <none> and password file <none>
I20240501 14:13:35.520407 1574 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:35.520558 1574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:35.526484 1574 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 14:13:35.529429 1588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:35.530895 1574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:35.531069 1574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0b5de4ca3a414e21ab0a898633a1bffd"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:35.531973 1574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:35.551752 1574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:35.552690 1574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:35.552994 1574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:35.554015 1574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:35.555764 1574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:35.555847 1574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:35.555987 1574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:35.556049 1574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:35.590932 1574 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.131:45217
I20240501 14:13:35.590955 1700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.131:45217 every 8 connection(s)
I20240501 14:13:35.592489 1574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:13:35.593948 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 1574
I20240501 14:13:35.594874 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:33275
--local_ip_for_outbound_sockets=127.16.36.132
--tserver_master_addrs=127.16.36.190:43297
--webserver_port=34443
--webserver_interface=127.16.36.132
--builtin_ntp_servers=127.16.36.148:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 14:13:35.607976 1701 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43297
I20240501 14:13:35.608364 1701 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:35.609349 1701 heartbeater.cc:511] Master 127.16.36.190:43297 requested a full tablet report, sending...
I20240501 14:13:35.611272 823 ts_manager.cc:194] Re-registered known tserver with Master: 0b5de4ca3a414e21ab0a898633a1bffd (127.16.36.131:45217)
I20240501 14:13:35.612198 823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.131:55021
W20240501 14:13:35.965395 1705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:35.965725 1705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:35.965873 1705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:35.973752 1705 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 14:13:35.973918 1705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:35.974089 1705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 14:13:35.983556 1705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:33275
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=34443
--tserver_master_addrs=127.16.36.190:43297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:35.984989 1705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:35.986878 1705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:35.999317 1711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:35.999807 1713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:36.000483 1710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:36.000991 1705 server_base.cc:1034] running on GCE node
I20240501 14:13:36.001674 1705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:36.002835 1705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:36.004096 1705 hybrid_clock.cc:648] HybridClock initialized: now 1714572816004049 us; error 50 us; skew 500 ppm
I20240501 14:13:36.004460 1705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:36.007166 1705 webserver.cc:460] Webserver started at http://127.16.36.132:34443/ using document root <none> and password file <none>
I20240501 14:13:36.007997 1705 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:36.008145 1705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:36.014201 1705 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 14:13:36.017472 1719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:36.019102 1705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:36.019299 1705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "922699286c8742c792604bd5db1378fb"
format_stamp: "Formatted at 2024-05-01 14:13:27 on dist-test-slave-9q9t"
I20240501 14:13:36.020238 1705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:36.046151 1705 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:36.047113 1705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:36.047410 1705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:36.048396 1705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:36.049988 1705 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:36.050069 1705 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:36.050205 1705 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:36.050261 1705 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:36.083853 1705 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:33275
I20240501 14:13:36.083897 1831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:33275 every 8 connection(s)
I20240501 14:13:36.085354 1705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:13:36.086819 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 1705
I20240501 14:13:36.106000 1832 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43297
I20240501 14:13:36.106437 1832 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:36.107396 1832 heartbeater.cc:511] Master 127.16.36.190:43297 requested a full tablet report, sending...
I20240501 14:13:36.109004 823 ts_manager.cc:194] Re-registered known tserver with Master: 922699286c8742c792604bd5db1378fb (127.16.36.132:33275)
I20240501 14:13:36.110266 823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:35719
I20240501 14:13:36.614180 1075 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:36.614773 1701 heartbeater.cc:503] Master 127.16.36.190:43297 was elected leader, sending a full tablet report...
I20240501 14:13:36.630151 1636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:36.641229 1767 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:36.644513 944 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:36.819082 1636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:36.823009 1075 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:36.831549 1767 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:36.834679 944 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.017563 1636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:37.020676 1075 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:13:37.022920 1767 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:37.031639 944 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.113212 1832 heartbeater.cc:503] Master 127.16.36.190:43297 was elected leader, sending a full tablet report...
I20240501 14:13:37.307968 1636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:37.314023 1767 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:13:37.316995 944 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:13:37.321938 1075 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d4cf24e21e0546beb7c379f1a0034d52 | 127.16.36.190:43297 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.148:39191 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0b5de4ca3a414e21ab0a898633a1bffd | 127.16.36.131:45217 | HEALTHY | <none> | 0 | 0
51efb78072074a22939c0b33c86b64f5 | 127.16.36.130:38333 | HEALTHY | <none> | 7 | 0
922699286c8742c792604bd5db1378fb | 127.16.36.132:33275 | HEALTHY | <none> | 0 | 0
9cd259d813db47c89bcb8b6f55a87fc7 | 127.16.36.129:38425 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.16.36.129 | experimental | 127.16.36.129:38425
local_ip_for_outbound_sockets | 127.16.36.130 | experimental | 127.16.36.130:38333
local_ip_for_outbound_sockets | 127.16.36.131 | experimental | 127.16.36.131:45217
local_ip_for_outbound_sockets | 127.16.36.132 | experimental | 127.16.36.132:33275
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.36.129:38425
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.36.130:38333
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.36.131:45217
server_dump_info_path | /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.36.132:33275
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.148:39191 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:13:37.372175 16530 log_verifier.cc:126] Checking tablet 30181ff3c42b4d5c8315daa414f170bf
I20240501 14:13:37.381429 16530 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 30181ff3c42b4d5c8315daa414f170bf
I20240501 14:13:37.381543 16530 log_verifier.cc:126] Checking tablet 3b9f9d396afa448bb979788eac3f7f86
I20240501 14:13:37.389519 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3b9f9d396afa448bb979788eac3f7f86
I20240501 14:13:37.389634 16530 log_verifier.cc:126] Checking tablet 45303dac715e4c83b9a24b7bc2db9236
I20240501 14:13:37.397900 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 45303dac715e4c83b9a24b7bc2db9236
I20240501 14:13:37.398001 16530 log_verifier.cc:126] Checking tablet 50b04bc92b67435c8d429b70224514ee
I20240501 14:13:37.406250 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 50b04bc92b67435c8d429b70224514ee
I20240501 14:13:37.406366 16530 log_verifier.cc:126] Checking tablet 52531661b6e145c99569da0409a82a56
I20240501 14:13:37.414721 16530 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 52531661b6e145c99569da0409a82a56
I20240501 14:13:37.414824 16530 log_verifier.cc:126] Checking tablet 5e4504b3feee486086d3cbc5701ed520
I20240501 14:13:37.422760 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5e4504b3feee486086d3cbc5701ed520
I20240501 14:13:37.422865 16530 log_verifier.cc:126] Checking tablet 5eb30806c2284c4594cbe4d62c49e001
I20240501 14:13:37.431078 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5eb30806c2284c4594cbe4d62c49e001
I20240501 14:13:37.431186 16530 log_verifier.cc:126] Checking tablet 68391462ac6d4a20bdf52c3d838cc0c7
I20240501 14:13:37.439730 16530 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 68391462ac6d4a20bdf52c3d838cc0c7
I20240501 14:13:37.439831 16530 log_verifier.cc:126] Checking tablet 6d6e38956cdd4e1897f70cf7e53d0aec
I20240501 14:13:37.447955 16530 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 6d6e38956cdd4e1897f70cf7e53d0aec
I20240501 14:13:37.448069 16530 log_verifier.cc:126] Checking tablet 74d6e9743fe64685a3c485c898e933fa
I20240501 14:13:37.455962 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 74d6e9743fe64685a3c485c898e933fa
I20240501 14:13:37.456071 16530 log_verifier.cc:126] Checking tablet 81d570b279fa4b9baaa6fb4d6fb9578c
I20240501 14:13:37.464332 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 81d570b279fa4b9baaa6fb4d6fb9578c
I20240501 14:13:37.464448 16530 log_verifier.cc:126] Checking tablet 8494e8aca92e4620b605d4ee2d0969e9
I20240501 14:13:37.472290 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8494e8aca92e4620b605d4ee2d0969e9
I20240501 14:13:37.472395 16530 log_verifier.cc:126] Checking tablet e5904c475ab54bd3b358115808705b8d
I20240501 14:13:37.480233 16530 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e5904c475ab54bd3b358115808705b8d
I20240501 14:13:37.481251 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 883
I20240501 14:13:37.482574 1004 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:37.589224 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 883
I20240501 14:13:37.618908 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 1014
I20240501 14:13:37.620039 1135 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:37.724273 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 1014
I20240501 14:13:37.746656 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 1574
I20240501 14:13:37.753286 1695 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:37.846421 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 1574
I20240501 14:13:37.866205 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 1705
I20240501 14:13:37.872408 1827 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:37.965873 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 1705
I20240501 14:13:37.985767 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 793
I20240501 14:13:37.986961 854 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:38.108901 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 793
2024-05-01T14:13:38Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (13111 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (13111 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240501 14:13:38.170222 16530 test_util.cc:268] Using random seed: 398770513
I20240501 14:13:38.172245 16530 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:13:38.172322 16530 ts_itest-base.cc:113] --------------
I20240501 14:13:38.172374 16530 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:13:38.172426 16530 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:13:38.172487 16530 ts_itest-base.cc:116] --------------
2024-05-01T14:13:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:13:38Z Disabled control of system clock
I20240501 14:13:38.237946 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:43879
--webserver_interface=127.16.36.190
--webserver_port=0
--builtin_ntp_servers=127.16.36.148:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.190:43879
--location_mapping_cmd=/tmp/dist-test-task1g_oHS/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 14:13:38.603188 1982 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:13:38.603564 1982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:38.603685 1982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:38.603799 1982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:38.611486 1982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:13:38.611651 1982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:13:38.611769 1982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:38.611840 1982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:13:38.611933 1982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:13:38.621457 1982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task1g_oHS/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.16.36.190:43879
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.190:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:38.622889 1982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:38.624656 1982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:38.635241 1988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:38.635973 1987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:38.636924 1982 server_base.cc:1034] running on GCE node
W20240501 14:13:38.637391 1990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:38.638273 1982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:38.639806 1982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:38.641057 1982 hybrid_clock.cc:648] HybridClock initialized: now 1714572818640989 us; error 47 us; skew 500 ppm
I20240501 14:13:38.641402 1982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:38.643806 1982 webserver.cc:460] Webserver started at http://127.16.36.190:37867/ using document root <none> and password file <none>
I20240501 14:13:38.644542 1982 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:38.644688 1982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:38.645058 1982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:38.648195 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8a923ddb15aa4b88a206914730d4ea0f"
format_stamp: "Formatted at 2024-05-01 14:13:38 on dist-test-slave-9q9t"
I20240501 14:13:38.649042 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8a923ddb15aa4b88a206914730d4ea0f"
format_stamp: "Formatted at 2024-05-01 14:13:38 on dist-test-slave-9q9t"
I20240501 14:13:38.654760 1982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 14:13:38.658394 1996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:38.659838 1982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:38.660015 1982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8a923ddb15aa4b88a206914730d4ea0f"
format_stamp: "Formatted at 2024-05-01 14:13:38 on dist-test-slave-9q9t"
I20240501 14:13:38.660200 1982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:38.676359 1982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:38.677244 1982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:38.677557 1982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:38.695137 1982 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.190:43879
I20240501 14:13:38.695170 2047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.190:43879 every 8 connection(s)
I20240501 14:13:38.696666 1982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:13:38.700588 2048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:13:38.706458 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 1982
I20240501 14:13:38.706737 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:13:38.711477 2048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f: Bootstrap starting.
I20240501 14:13:38.715739 2048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f: Neither blocks nor log segments found. Creating new log.
I20240501 14:13:38.717101 2048 log.cc:824] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f: Log is configured to *not* fsync() on all Append() calls
I20240501 14:13:38.720640 2048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f: No bootstrap required, opened a new log
I20240501 14:13:38.727761 2048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a923ddb15aa4b88a206914730d4ea0f" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43879 } }
I20240501 14:13:38.728135 2048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:13:38.728245 2048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a923ddb15aa4b88a206914730d4ea0f, State: Initialized, Role: FOLLOWER
I20240501 14:13:38.728801 2048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a923ddb15aa4b88a206914730d4ea0f" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43879 } }
I20240501 14:13:38.728994 2048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:13:38.729092 2048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:13:38.729216 2048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:13:38.731765 2048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a923ddb15aa4b88a206914730d4ea0f" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43879 } }
I20240501 14:13:38.732234 2048 leader_election.cc:302] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a923ddb15aa4b88a206914730d4ea0f; no voters:
I20240501 14:13:38.732800 2048 leader_election.cc:288] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:13:38.732942 2053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:13:38.733597 2053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [term 1 LEADER]: Becoming Leader. State: Replica: 8a923ddb15aa4b88a206914730d4ea0f, State: Running, Role: LEADER
I20240501 14:13:38.734195 2053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a923ddb15aa4b88a206914730d4ea0f" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43879 } }
I20240501 14:13:38.734972 2048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:13:38.737306 2055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a923ddb15aa4b88a206914730d4ea0f. Latest consensus state: current_term: 1 leader_uuid: "8a923ddb15aa4b88a206914730d4ea0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a923ddb15aa4b88a206914730d4ea0f" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43879 } } }
I20240501 14:13:38.737730 2055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [sys.catalog]: This master's current role is: LEADER
I20240501 14:13:38.737380 2054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a923ddb15aa4b88a206914730d4ea0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a923ddb15aa4b88a206914730d4ea0f" member_type: VOTER last_known_addr { host: "127.16.36.190" port: 43879 } } }
I20240501 14:13:38.739495 2054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f [sys.catalog]: This master's current role is: LEADER
I20240501 14:13:38.740336 2059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:13:38.747133 2059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:13:38.756072 2059 catalog_manager.cc:1347] Generated new cluster ID: fc3bef8d40534fbba566310a4fab24d0
I20240501 14:13:38.756202 2059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:13:38.766888 2059 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:13:38.768165 2059 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:13:38.779914 2059 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8a923ddb15aa4b88a206914730d4ea0f: Generated new TSK 0
I20240501 14:13:38.780833 2059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:13:38.787142 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.129:0
--local_ip_for_outbound_sockets=127.16.36.129
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--builtin_ntp_servers=127.16.36.148:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:13:39.154898 2072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:39.155217 2072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:39.155346 2072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:39.163214 2072 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:13:39.163347 2072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:39.163519 2072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.129
I20240501 14:13:39.173377 2072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.129
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.129
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:39.174821 2072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:39.176685 2072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:39.189651 2078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:39.190150 2080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:39.190855 2077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:39.190989 2072 server_base.cc:1034] running on GCE node
I20240501 14:13:39.191659 2072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:39.192665 2072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:39.193872 2072 hybrid_clock.cc:648] HybridClock initialized: now 1714572819193825 us; error 48 us; skew 500 ppm
I20240501 14:13:39.194196 2072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:39.196548 2072 webserver.cc:460] Webserver started at http://127.16.36.129:44279/ using document root <none> and password file <none>
I20240501 14:13:39.197306 2072 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:39.197435 2072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:39.197780 2072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:39.200507 2072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "17d4e2007a5743dc8378e0ca0253e805"
format_stamp: "Formatted at 2024-05-01 14:13:39 on dist-test-slave-9q9t"
I20240501 14:13:39.201304 2072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "17d4e2007a5743dc8378e0ca0253e805"
format_stamp: "Formatted at 2024-05-01 14:13:39 on dist-test-slave-9q9t"
I20240501 14:13:39.206660 2072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 14:13:39.210139 2086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:39.211549 2072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:39.211740 2072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "17d4e2007a5743dc8378e0ca0253e805"
format_stamp: "Formatted at 2024-05-01 14:13:39 on dist-test-slave-9q9t"
I20240501 14:13:39.212071 2072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:39.236757 2072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:39.237648 2072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:39.237927 2072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:39.238965 2072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:39.240693 2072 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:39.240795 2072 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:39.240913 2072 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:39.240993 2072 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:39.274991 2072 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.129:40097
I20240501 14:13:39.275066 2198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.129:40097 every 8 connection(s)
I20240501 14:13:39.276345 2072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:13:39.279543 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 2072
I20240501 14:13:39.279778 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:13:39.283298 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.130:0
--local_ip_for_outbound_sockets=127.16.36.130
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--builtin_ntp_servers=127.16.36.148:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:13:39.290261 2199 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43879
I20240501 14:13:39.290575 2199 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:39.291492 2199 heartbeater.cc:511] Master 127.16.36.190:43879 requested a full tablet report, sending...
I20240501 14:13:39.341015 2013 ts_manager.cc:194] Registered new tserver with Master: 17d4e2007a5743dc8378e0ca0253e805 (127.16.36.129:40097)
I20240501 14:13:39.343068 2013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.129:38775
W20240501 14:13:39.646009 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:39.646343 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:39.646485 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:39.654264 2203 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:13:39.654423 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:39.654582 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.130
I20240501 14:13:39.663895 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.130
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.130
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:39.665319 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:39.667145 2203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:39.679621 2209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:39.680249 2210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:39.680752 2212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:39.681120 2203 server_base.cc:1034] running on GCE node
I20240501 14:13:39.681739 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:39.682775 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:39.684003 2203 hybrid_clock.cc:648] HybridClock initialized: now 1714572819683948 us; error 43 us; skew 500 ppm
I20240501 14:13:39.684331 2203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:39.686679 2203 webserver.cc:460] Webserver started at http://127.16.36.130:35149/ using document root <none> and password file <none>
I20240501 14:13:39.687462 2203 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:39.687588 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:39.687920 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:39.690609 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1cc708aef17a498ca22d665643c8f601"
format_stamp: "Formatted at 2024-05-01 14:13:39 on dist-test-slave-9q9t"
I20240501 14:13:39.691555 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1cc708aef17a498ca22d665643c8f601"
format_stamp: "Formatted at 2024-05-01 14:13:39 on dist-test-slave-9q9t"
I20240501 14:13:39.697118 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:13:39.700678 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:39.702075 2203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:13:39.702232 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1cc708aef17a498ca22d665643c8f601"
format_stamp: "Formatted at 2024-05-01 14:13:39 on dist-test-slave-9q9t"
I20240501 14:13:39.702442 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:39.717563 2203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:39.718461 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:39.718786 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:39.719818 2203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:39.721410 2203 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:39.721506 2203 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:39.721632 2203 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:39.721706 2203 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:39.754842 2203 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.130:39719
I20240501 14:13:39.754902 2330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.130:39719 every 8 connection(s)
I20240501 14:13:39.756232 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:13:39.761365 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 2203
I20240501 14:13:39.761560 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:13:39.765318 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.131:0
--local_ip_for_outbound_sockets=127.16.36.131
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--builtin_ntp_servers=127.16.36.148:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:13:39.770136 2331 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43879
I20240501 14:13:39.770457 2331 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:39.775092 2331 heartbeater.cc:511] Master 127.16.36.190:43879 requested a full tablet report, sending...
I20240501 14:13:39.842408 2013 ts_manager.cc:194] Registered new tserver with Master: 1cc708aef17a498ca22d665643c8f601 (127.16.36.130:39719)
I20240501 14:13:39.843466 2013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.130:38819
W20240501 14:13:40.146806 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:40.147117 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:40.147259 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:40.155212 2334 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:13:40.155355 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:40.155534 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.131
I20240501 14:13:40.165051 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.131
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.131
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:40.166401 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:40.168176 2334 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:40.180794 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:40.181388 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:40.181843 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:40.182498 2334 server_base.cc:1034] running on GCE node
I20240501 14:13:40.183286 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:40.184281 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:40.185501 2334 hybrid_clock.cc:648] HybridClock initialized: now 1714572820185448 us; error 46 us; skew 500 ppm
I20240501 14:13:40.185859 2334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:40.188252 2334 webserver.cc:460] Webserver started at http://127.16.36.131:38687/ using document root <none> and password file <none>
I20240501 14:13:40.189055 2334 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:40.189182 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:40.189517 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:40.192487 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f381951284524cc5a5b4f5a383b31bb7"
format_stamp: "Formatted at 2024-05-01 14:13:40 on dist-test-slave-9q9t"
I20240501 14:13:40.193367 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f381951284524cc5a5b4f5a383b31bb7"
format_stamp: "Formatted at 2024-05-01 14:13:40 on dist-test-slave-9q9t"
I20240501 14:13:40.198948 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20240501 14:13:40.202811 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:40.204357 2334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:13:40.204535 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f381951284524cc5a5b4f5a383b31bb7"
format_stamp: "Formatted at 2024-05-01 14:13:40 on dist-test-slave-9q9t"
I20240501 14:13:40.204764 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:40.218676 2334 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:40.219609 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:40.219902 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:40.220930 2334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:40.222553 2334 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:40.222688 2334 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:40.222815 2334 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:40.222901 2334 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:40.256278 2334 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.131:46231
I20240501 14:13:40.256357 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.131:46231 every 8 connection(s)
I20240501 14:13:40.257654 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:13:40.264276 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 2334
I20240501 14:13:40.264441 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:13:40.267849 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.132:0
--local_ip_for_outbound_sockets=127.16.36.132
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--builtin_ntp_servers=127.16.36.148:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:13:40.271250 2462 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43879
I20240501 14:13:40.271624 2462 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:40.272619 2462 heartbeater.cc:511] Master 127.16.36.190:43879 requested a full tablet report, sending...
I20240501 14:13:40.324852 2013 ts_manager.cc:194] Registered new tserver with Master: f381951284524cc5a5b4f5a383b31bb7 (127.16.36.131:46231)
I20240501 14:13:40.326218 2013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.131:50305
I20240501 14:13:40.346284 2199 heartbeater.cc:503] Master 127.16.36.190:43879 was elected leader, sending a full tablet report...
W20240501 14:13:40.641037 2466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:40.641360 2466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:40.641530 2466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:40.649652 2466 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:13:40.649803 2466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:40.649986 2466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.132
I20240501 14:13:40.659782 2466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.132
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.132
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:40.661278 2466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:40.663226 2466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:40.676131 2471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:40.677053 2472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:40.677690 2474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:40.677944 2466 server_base.cc:1034] running on GCE node
I20240501 14:13:40.678580 2466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:40.679658 2466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:40.680860 2466 hybrid_clock.cc:648] HybridClock initialized: now 1714572820680794 us; error 59 us; skew 500 ppm
I20240501 14:13:40.681169 2466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:40.683626 2466 webserver.cc:460] Webserver started at http://127.16.36.132:39579/ using document root <none> and password file <none>
I20240501 14:13:40.684484 2466 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:40.684602 2466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:40.684877 2466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:40.687546 2466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "11f4119a22154aec9582cb2d2d274d74"
format_stamp: "Formatted at 2024-05-01 14:13:40 on dist-test-slave-9q9t"
I20240501 14:13:40.688371 2466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "11f4119a22154aec9582cb2d2d274d74"
format_stamp: "Formatted at 2024-05-01 14:13:40 on dist-test-slave-9q9t"
I20240501 14:13:40.693917 2466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 14:13:40.698364 2480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:40.700008 2466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:40.700187 2466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "11f4119a22154aec9582cb2d2d274d74"
format_stamp: "Formatted at 2024-05-01 14:13:40 on dist-test-slave-9q9t"
I20240501 14:13:40.700424 2466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:40.719146 2466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:40.720048 2466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:40.720297 2466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:40.721261 2466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:40.722884 2466 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:40.722978 2466 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:40.723059 2466 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:40.723114 2466 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:40.754324 2466 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.132:33767
I20240501 14:13:40.754393 2592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.132:33767 every 8 connection(s)
I20240501 14:13:40.756151 2466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:13:40.762989 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 2466
I20240501 14:13:40.763186 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:13:40.767031 16530 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1g_oHS/build/asan/bin/kudu
/tmp/dist-test-task1g_oHS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.133:0
--local_ip_for_outbound_sockets=127.16.36.133
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--builtin_ntp_servers=127.16.36.148:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:13:40.770447 2593 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43879
I20240501 14:13:40.770834 2593 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:40.771756 2593 heartbeater.cc:511] Master 127.16.36.190:43879 requested a full tablet report, sending...
I20240501 14:13:40.827903 2013 ts_manager.cc:194] Registered new tserver with Master: 11f4119a22154aec9582cb2d2d274d74 (127.16.36.132:33767)
I20240501 14:13:40.828898 2013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.132:36617
I20240501 14:13:40.846324 2331 heartbeater.cc:503] Master 127.16.36.190:43879 was elected leader, sending a full tablet report...
W20240501 14:13:41.132552 2597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:13:41.132879 2597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:13:41.133026 2597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:13:41.140820 2597 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:13:41.140962 2597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:13:41.141136 2597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.133
I20240501 14:13:41.150538 2597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.148:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.133
--webserver_port=0
--tserver_master_addrs=127.16.36.190:43879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.133
--log_dir=/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:13:41.151989 2597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:13:41.153725 2597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:13:41.165941 2603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:41.166357 2602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:13:41.166998 2605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:13:41.167156 2597 server_base.cc:1034] running on GCE node
I20240501 14:13:41.167785 2597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:13:41.168759 2597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:13:41.169962 2597 hybrid_clock.cc:648] HybridClock initialized: now 1714572821169915 us; error 41 us; skew 500 ppm
I20240501 14:13:41.170301 2597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:13:41.172653 2597 webserver.cc:460] Webserver started at http://127.16.36.133:36121/ using document root <none> and password file <none>
I20240501 14:13:41.173400 2597 fs_manager.cc:359] Metadata directory not provided
I20240501 14:13:41.173524 2597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:13:41.173833 2597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:13:41.176513 2597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "910dfaf2dfb14def9f8a6442b70d7cca"
format_stamp: "Formatted at 2024-05-01 14:13:41 on dist-test-slave-9q9t"
I20240501 14:13:41.177292 2597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "910dfaf2dfb14def9f8a6442b70d7cca"
format_stamp: "Formatted at 2024-05-01 14:13:41 on dist-test-slave-9q9t"
I20240501 14:13:41.182760 2597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:13:41.186422 2611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:41.187813 2597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:13:41.187987 2597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "910dfaf2dfb14def9f8a6442b70d7cca"
format_stamp: "Formatted at 2024-05-01 14:13:41 on dist-test-slave-9q9t"
I20240501 14:13:41.188202 2597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:13:41.233886 2597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:13:41.234855 2597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:13:41.235136 2597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:13:41.236140 2597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:13:41.237900 2597 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:13:41.238000 2597 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:41.238117 2597 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:13:41.238189 2597 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:13:41.275975 2597 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.133:36607
I20240501 14:13:41.276034 2723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.133:36607 every 8 connection(s)
I20240501 14:13:41.277508 2597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:13:41.282496 16530 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1g_oHS/build/asan/bin/kudu as pid 2597
I20240501 14:13:41.282747 16530 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1g_oHS/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714571953216231-16530-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:13:41.291885 2724 heartbeater.cc:347] Connected to a master server at 127.16.36.190:43879
I20240501 14:13:41.292210 2724 heartbeater.cc:464] Registering TS with master...
I20240501 14:13:41.293108 2724 heartbeater.cc:511] Master 127.16.36.190:43879 requested a full tablet report, sending...
I20240501 14:13:41.329064 2462 heartbeater.cc:503] Master 127.16.36.190:43879 was elected leader, sending a full tablet report...
I20240501 14:13:41.343580 2013 ts_manager.cc:194] Registered new tserver with Master: 910dfaf2dfb14def9f8a6442b70d7cca (127.16.36.133:36607)
I20240501 14:13:41.344348 16530 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:13:41.344659 2013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.133:59403
I20240501 14:13:41.413492 16530 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:13:41.413632 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 2597
I20240501 14:13:41.422600 2719 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:41.529271 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 2597
I20240501 14:13:41.547307 16530 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1g_oHS/build/asan/bin/kudu pid 2466
I20240501 14:13:41.560290 2588 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:13:41.641278 16530 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1g_oHS/build/asan/bin/kudu with pid 2466
************************ BEGIN STACKS **************************
[New LWP 16533]
[New LWP 16534]
[New LWP 1975]
[New LWP 1976]
[New LWP 1977]
[New LWP 1978]
[New LWP 1979]
[New LWP 2729]
[New LWP 2730]
[New LWP 2731]
[New LWP 2732]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fcfd831ad50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7fcfcd56bd80 (LWP 16530) "rebalancer_tool" 0x00007fcfd831ad50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7fcfc90bd700 (LWP 16533) "rebalancer_tool" 0x00007fcfd831b32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7fcfc70c4700 (LWP 16534) "kernel-watcher-" 0x00007fcfd8316fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7fcfc68c3700 (LWP 1975) "rpc reactor-197" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7fcfc48bf700 (LWP 1976) "rpc reactor-197" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7fcfc60c2700 (LWP 1977) "rpc reactor-197" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7fcfc3696700 (LWP 1978) "rpc reactor-197" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7fcfc519f700 (LWP 1979) "rpc reactor-197" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7fcfc2e95700 (LWP 2729) "rpc reactor-272" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7fcfc2694700 (LWP 2730) "rpc reactor-273" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7fcfc16a2700 (LWP 2731) "rpc reactor-273" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7fcfc0ea1700 (LWP 2732) "rpc reactor-273" 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7fcfc0ea1700 (LWP 2732)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x6190016f6280, timeout=0.001) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x6190016f6280, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x6140000514a0, flags=18944, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x614000051498) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x611000912618) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7fcfc16a2700 (LWP 2731)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x6190016f5d80, timeout=0.001) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x6190016f5d80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x6140000512a0, flags=130944, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x614000051298) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x611000912118) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7fcfc2694700 (LWP 2730)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x6190016f5880, timeout=0.099634245997549442) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x6190016f5880, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x6140000510a0, flags=132736, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x614000051098) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x6110009124d8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7fcfc2e95700 (LWP 2729)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x6190016f5380, timeout=0.099541586997474951) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x6190016f5380, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x614000050ea0, flags=131840, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x614000050e98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x611000911ad8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7fcfc519f700 (LWP 1979)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x619000a9bf80, timeout=0.09965517499449561) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x619000a9bf80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x61400006f2a0, flags=196352, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x61400006f298) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x6110008f2958) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7fcfc3696700 (LWP 1978)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x619000a9f180, timeout=0.099105023994525254) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x619000a9f180, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x61400006eca0, flags=198144, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x61400006ec98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x6110008f3998) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7fcfc60c2700 (LWP 1977)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x619000af0580, timeout=0.09904511299441765) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x619000af0580, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x61400006eaa0, flags=197248, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x61400006ea98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x6110008f3ad8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7fcfc48bf700 (LWP 1976)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x619000af0a80, timeout=0.099022278994652879) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x619000af0a80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x61400006e8a0, flags=199040, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x61400006e898) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x6110008f3c18) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7fcfc68c3700 (LWP 1975)):
#0 0x00007fcfce106a47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcfcfd80de0 in epoll_poll (loop=0x619000af0f80, timeout=0.099554039994472987) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007fcfcfd8449e in ev_run (loop=0x619000af0f80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007fcfd41927c1 in ev::loop_ref::run (this=this@entry=0x61400006e6a0, flags=199936, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007fcfd418ac16 in kudu::rpc::ReactorThread::RunThread (this=0x61400006e698) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007fcfd418fd3c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007fcfd418fbb9 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x6110008f3d58) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7fcfc70c4700 (LWP 16534)):
#0 0x00007fcfd8316fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fcfd1bd53ae in kudu::ConditionVariable::WaitUntil (this=0x612000021b98, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/condition_variable.cc:87
#2 0x00000000006c4dac in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:87
#3 0x00000000006c4c3a in kudu::CountDownLatch::WaitFor (this=0x612000021bc0, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:97
#4 0x00007fcfd1d0c48f in kudu::KernelStackWatchdog::RunThread (this=0x612000021ac0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007fcfd1d0d45c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007fcfd1d0d2d9 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007fcfe65a398d in std::function<void ()>::operator()() const (this=0x611000057d98) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007fcfd1e6701e in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694
#9 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7fcfc90bd700 (LWP 16533)):
#0 0x00007fcfd831b32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000053a46f in __interceptor_waitpid (pid=<optimized out>, status=0x7fcfc90bb6b0, options=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007fcfd1e515b6 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:817
#3 0x00007fcfd1e512e5 in kudu::Subprocess::Wait (this=0x7fcfc90bb6b0, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:544
#4 0x00007fcfd1ce6638 in kudu::PstackWatcher::RunStackDump (argv=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:258
#5 0x00007fcfd1ce5d14 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:218
#6 0x00007fcfd1ce51a9 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:169
#7 0x00007fcfd1ce3537 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:162
#8 0x00007fcfe645663b in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:60
#9 0x00007fcfe64564d3 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#10 0x00007fcfe6456453 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
#11 0x00007fcfe6456429 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x60200000d298) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#12 0x00007fcfe64563e9 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x60200000d298) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#13 0x00007fcfe64562eb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
#14 0x00007fcfce6ad6df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007fcfd83106db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007fcfce10671f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7fcfcd56bd80 (LWP 16530)):
#0 0x00007fcfd831ad50 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fcfd0cd7f5b in base::SleepForNanoseconds (nanoseconds=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/gutil/sysinfo.cc:96
#2 0x00007fcfd1da1d16 in kudu::SleepFor (delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/monotime.cc:264
#3 0x000000000064df0f in kudu::tools::RebalancingTest::CreateTablesExcludingLocations (this=0x619000af2380, excluded_locations=..., table_name_pattern=..., table_names=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:834
#4 0x00000000006412b6 in kudu::tools::RebalancingTest::Prepare (this=<optimized out>, extra_tserver_flags=..., extra_master_flags=..., location_info=..., empty_locations=..., created_tables_names=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:798
#5 0x00000000006129cf in kudu::tools::LocationAwareRebalancingParamTest_Rebalance_Test::TestBody (this=0x619000af2380) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:2115
#6 0x00007fcfe5f0b0ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7fcfe5f0c82b "the test body", method=<optimized out>, object=0x619000af2380) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#7 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x619000af2380, method=<optimized out>, location=location@entry=0x7fcfe5f0c82b "the test body") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#8 0x00007fcfe5effbdb in testing::Test::Run (this=0x619000af2380) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#9 0x00007fcfe5effd9d in testing::TestInfo::Run (this=0x612000007b40) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#10 0x00007fcfe5f00377 in testing::TestSuite::Run (this=0x612000007840) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#11 0x00007fcfe5f0077c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#12 0x00007fcfe5f0b60d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fcfe5f0e3e8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x616000006080) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#13 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x616000006080, method=<optimized out>, location=location@entry=0x7fcfe5f0e3e8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#14 0x00007fcfe5effe63 in testing::UnitTest::Run (this=0x7fcfe611f1e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#15 0x00007fcfe64573c2 in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#16 0x00007fcfe64558d9 in main (argc=<optimized out>, argv=0x7ffcfe0fa3f8) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109
#17 0x00007fcfce006bf7 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x00000000005023da in _start ()
I20240501 14:14:10.688400 2013 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40818:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 14:14:10.695645 2013 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:14:10.752473 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 507ea0f416f3403780bab36c07797f77 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.751999 2266 tablet_service.cc:1467] Processing CreateTablet for tablet fe0cd858c23f481087a4e4b12eb42fe8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.753199 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 37be6bb30d58488da944702da0c12784 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.753687 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507ea0f416f3403780bab36c07797f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.753705 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0cd858c23f481087a4e4b12eb42fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.754791 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37be6bb30d58488da944702da0c12784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.755133 2132 tablet_service.cc:1467] Processing CreateTablet for tablet 37be6bb30d58488da944702da0c12784 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.755689 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37be6bb30d58488da944702da0c12784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.751999 2134 tablet_service.cc:1467] Processing CreateTablet for tablet fe0cd858c23f481087a4e4b12eb42fe8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.756165 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0cd858c23f481087a4e4b12eb42fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.758134 2397 tablet_service.cc:1467] Processing CreateTablet for tablet fe0cd858c23f481087a4e4b12eb42fe8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.758859 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 507ea0f416f3403780bab36c07797f77 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.759411 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507ea0f416f3403780bab36c07797f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.759601 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0cd858c23f481087a4e4b12eb42fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.758025 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 37be6bb30d58488da944702da0c12784 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.763089 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37be6bb30d58488da944702da0c12784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.762934 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 507ea0f416f3403780bab36c07797f77 (DEFAULT_TABLE table=rebalance_test_table_0 [id=eef0f8e6b00246fb8473bea652d726d6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:14:10.763932 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507ea0f416f3403780bab36c07797f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:10.791201 2757 tablet_bootstrap.cc:492] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805: Bootstrap starting.
I20240501 14:14:10.795981 2757 tablet_bootstrap.cc:654] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805: Neither blocks nor log segments found. Creating new log.
I20240501 14:14:10.797406 2757 log.cc:824] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805: Log is configured to *not* fsync() on all Append() calls
I20240501 14:14:10.801595 2758 tablet_bootstrap.cc:492] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7: Bootstrap starting.
I20240501 14:14:10.801837 2759 tablet_bootstrap.cc:492] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601: Bootstrap starting.
I20240501 14:14:10.806833 2759 tablet_bootstrap.cc:654] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601: Neither blocks nor log segments found. Creating new log.
I20240501 14:14:10.807222 2758 tablet_bootstrap.cc:654] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7: Neither blocks nor log segments found. Creating new log.
I20240501 14:14:10.808337 2759 log.cc:824] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601: Log is configured to *not* fsync() on all Append() calls
I20240501 14:14:10.808528 2758 log.cc:824] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:14:10.810055 2757 tablet_bootstrap.cc:492] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805: No bootstrap required, opened a new log
I20240501 14:14:10.810561 2757 ts_tablet_manager.cc:1367] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20240501 14:14:10.817909 2758 tablet_bootstrap.cc:492] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7: No bootstrap required, opened a new log
I20240501 14:14:10.818289 2758 ts_tablet_manager.cc:1367] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240501 14:14:10.822387 2757 raft_consensus.cc:348] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.822896 2757 raft_consensus.cc:374] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.823014 2757 raft_consensus.cc:729] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d4e2007a5743dc8378e0ca0253e805, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.823802 2757 consensus_queue.cc:260] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.826476 2757 ts_tablet_manager.cc:1398] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20240501 14:14:10.826989 2757 tablet_bootstrap.cc:492] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805: Bootstrap starting.
I20240501 14:14:10.827023 2759 tablet_bootstrap.cc:492] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601: No bootstrap required, opened a new log
I20240501 14:14:10.827343 2759 ts_tablet_manager.cc:1367] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.011s
I20240501 14:14:10.829146 2757 tablet_bootstrap.cc:654] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805: Neither blocks nor log segments found. Creating new log.
I20240501 14:14:10.836777 2758 raft_consensus.cc:348] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.837384 2758 raft_consensus.cc:374] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.837531 2758 raft_consensus.cc:729] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f381951284524cc5a5b4f5a383b31bb7, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.837576 2757 tablet_bootstrap.cc:492] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805: No bootstrap required, opened a new log
I20240501 14:14:10.837368 2759 raft_consensus.cc:348] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.837756 2757 ts_tablet_manager.cc:1367] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240501 14:14:10.837818 2759 raft_consensus.cc:374] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.837922 2759 raft_consensus.cc:729] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc708aef17a498ca22d665643c8f601, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.838470 2758 consensus_queue.cc:260] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.838508 2759 consensus_queue.cc:260] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.839129 2757 raft_consensus.cc:348] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.839380 2757 raft_consensus.cc:374] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.839483 2757 raft_consensus.cc:729] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d4e2007a5743dc8378e0ca0253e805, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.839747 2757 consensus_queue.cc:260] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.840562 2757 ts_tablet_manager.cc:1398] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:14:10.840876 2757 tablet_bootstrap.cc:492] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805: Bootstrap starting.
I20240501 14:14:10.843004 2757 tablet_bootstrap.cc:654] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805: Neither blocks nor log segments found. Creating new log.
************************* END STACKS ***************************
F20240501 14:14:10.848590 16533 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1714572850 (unix time) try "date -d @1714572850" if you are using GNU date ***
I20240501 14:14:10.852043 2757 tablet_bootstrap.cc:492] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805: No bootstrap required, opened a new log
I20240501 14:14:10.852241 2757 ts_tablet_manager.cc:1367] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240501 14:14:10.852389 2758 ts_tablet_manager.cc:1398] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7: Time spent starting tablet: real 0.034s user 0.012s sys 0.003s
I20240501 14:14:10.853163 2757 raft_consensus.cc:348] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.853440 2757 raft_consensus.cc:374] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.853528 2757 raft_consensus.cc:729] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d4e2007a5743dc8378e0ca0253e805, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.853915 2757 consensus_queue.cc:260] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.854769 2757 ts_tablet_manager.cc:1398] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:14:10.855949 2758 tablet_bootstrap.cc:492] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7: Bootstrap starting.
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004092) received by PID 16530 (TID 0x7fcfc90bd700) from PID 16530; stack trace: ***
I20240501 14:14:10.858000 2758 tablet_bootstrap.cc:654] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7: Neither blocks nor log segments found. Creating new log.
@ 0x7fcfd831b980 (unknown) at ??:0
I20240501 14:14:10.862305 2759 ts_tablet_manager.cc:1398] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601: Time spent starting tablet: real 0.035s user 0.015s sys 0.017s
I20240501 14:14:10.862802 2759 tablet_bootstrap.cc:492] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601: Bootstrap starting.
@ 0x7fcfce023fb7 gsignal at ??:0
I20240501 14:14:10.864647 2759 tablet_bootstrap.cc:654] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601: Neither blocks nor log segments found. Creating new log.
I20240501 14:14:10.865849 2758 tablet_bootstrap.cc:492] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7: No bootstrap required, opened a new log
I20240501 14:14:10.866083 2758 ts_tablet_manager.cc:1367] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
@ 0x7fcfce025921 abort at ??:0
I20240501 14:14:10.867334 2758 raft_consensus.cc:348] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.867599 2758 raft_consensus.cc:374] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.867703 2758 raft_consensus.cc:729] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f381951284524cc5a5b4f5a383b31bb7, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.868044 2758 consensus_queue.cc:260] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.868716 2758 ts_tablet_manager.cc:1398] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:14:10.869164 2758 tablet_bootstrap.cc:492] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7: Bootstrap starting.
@ 0x7fcfd083ddcd google::LogMessage::Fail() at ??:0
I20240501 14:14:10.871048 2758 tablet_bootstrap.cc:654] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7: Neither blocks nor log segments found. Creating new log.
@ 0x7fcfd0841b93 google::LogMessage::SendToLog() at ??:0
I20240501 14:14:10.873174 2759 tablet_bootstrap.cc:492] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601: No bootstrap required, opened a new log
I20240501 14:14:10.873371 2759 ts_tablet_manager.cc:1367] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
@ 0x7fcfd083d7cc google::LogMessage::Flush() at ??:0
I20240501 14:14:10.874650 2759 raft_consensus.cc:348] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.874913 2759 raft_consensus.cc:374] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.875010 2759 raft_consensus.cc:729] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc708aef17a498ca22d665643c8f601, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.875370 2759 consensus_queue.cc:260] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.876068 2759 ts_tablet_manager.cc:1398] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
@ 0x7fcfd083ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:14:10.876403 2759 tablet_bootstrap.cc:492] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601: Bootstrap starting.
I20240501 14:14:10.878464 2759 tablet_bootstrap.cc:654] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601: Neither blocks nor log segments found. Creating new log.
@ 0x7fcfe64567d9 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at ??:0
I20240501 14:14:10.882617 2758 tablet_bootstrap.cc:492] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7: No bootstrap required, opened a new log
I20240501 14:14:10.882846 2758 ts_tablet_manager.cc:1367] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
@ 0x7fcfe64564d3 std::__invoke_impl<>() at ??:0
I20240501 14:14:10.883793 2758 raft_consensus.cc:348] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.884076 2758 raft_consensus.cc:374] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.884155 2758 raft_consensus.cc:729] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f381951284524cc5a5b4f5a383b31bb7, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.884516 2758 consensus_queue.cc:260] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.885321 2758 ts_tablet_manager.cc:1398] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:14:10.887068 2759 tablet_bootstrap.cc:492] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601: No bootstrap required, opened a new log
I20240501 14:14:10.887257 2759 ts_tablet_manager.cc:1367] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
@ 0x7fcfe6456453 std::__invoke<>() at ??:0
I20240501 14:14:10.888176 2759 raft_consensus.cc:348] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.888458 2759 raft_consensus.cc:374] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:14:10.888533 2759 raft_consensus.cc:729] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc708aef17a498ca22d665643c8f601, State: Initialized, Role: FOLLOWER
I20240501 14:14:10.888856 2759 consensus_queue.cc:260] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.889508 2759 ts_tablet_manager.cc:1398] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:14:10.890307 2763 raft_consensus.cc:482] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:10.890590 2763 raft_consensus.cc:504] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
@ 0x7fcfe6456429 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
I20240501 14:14:10.892982 2763 leader_election.cc:288] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc708aef17a498ca22d665643c8f601 (127.16.36.130:39719), f381951284524cc5a5b4f5a383b31bb7 (127.16.36.131:46231)
@ 0x7fcfe64563e9 std::thread::_Invoker<>::operator()() at ??:0
@ 0x7fcfe64562eb std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7fcfce6ad6df (unknown) at ??:0
I20240501 14:14:10.903637 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37be6bb30d58488da944702da0c12784" candidate_uuid: "17d4e2007a5743dc8378e0ca0253e805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc708aef17a498ca22d665643c8f601" is_pre_election: true
I20240501 14:14:10.903712 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37be6bb30d58488da944702da0c12784" candidate_uuid: "17d4e2007a5743dc8378e0ca0253e805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f381951284524cc5a5b4f5a383b31bb7" is_pre_election: true
I20240501 14:14:10.904152 2286 raft_consensus.cc:2454] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d4e2007a5743dc8378e0ca0253e805 in term 0.
I20240501 14:14:10.904247 2417 raft_consensus.cc:2454] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d4e2007a5743dc8378e0ca0253e805 in term 0.
@ 0x7fcfd83106db start_thread at ??:0
I20240501 14:14:10.905004 2090 leader_election.cc:302] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d4e2007a5743dc8378e0ca0253e805, f381951284524cc5a5b4f5a383b31bb7; no voters:
I20240501 14:14:10.905395 2763 raft_consensus.cc:2781] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:14:10.905555 2763 raft_consensus.cc:482] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:14:10.905665 2763 raft_consensus.cc:3037] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Advancing to term 1
@ 0x7fcfce10671f clone at ??:0
I20240501 14:14:10.908457 2763 raft_consensus.cc:504] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.909260 2763 leader_election.cc:288] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc708aef17a498ca22d665643c8f601 (127.16.36.130:39719), f381951284524cc5a5b4f5a383b31bb7 (127.16.36.131:46231)
I20240501 14:14:10.909798 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37be6bb30d58488da944702da0c12784" candidate_uuid: "17d4e2007a5743dc8378e0ca0253e805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc708aef17a498ca22d665643c8f601"
I20240501 14:14:10.909858 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37be6bb30d58488da944702da0c12784" candidate_uuid: "17d4e2007a5743dc8378e0ca0253e805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f381951284524cc5a5b4f5a383b31bb7"
I20240501 14:14:10.909986 2286 raft_consensus.cc:3037] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:14:10.910097 2417 raft_consensus.cc:3037] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:14:10.912503 2286 raft_consensus.cc:2454] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d4e2007a5743dc8378e0ca0253e805 in term 1.
I20240501 14:14:10.913475 2417 raft_consensus.cc:2454] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d4e2007a5743dc8378e0ca0253e805 in term 1.
I20240501 14:14:10.913861 2090 leader_election.cc:302] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d4e2007a5743dc8378e0ca0253e805, 1cc708aef17a498ca22d665643c8f601; no voters:
I20240501 14:14:10.914568 2763 raft_consensus.cc:2781] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:14:10.915146 2763 raft_consensus.cc:686] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [term 1 LEADER]: Becoming Leader. State: Replica: 17d4e2007a5743dc8378e0ca0253e805, State: Running, Role: LEADER
I20240501 14:14:10.915817 2763 consensus_queue.cc:237] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.921801 2013 catalog_manager.cc:5484] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d4e2007a5743dc8378e0ca0253e805 (127.16.36.129). New cstate: current_term: 1 leader_uuid: "17d4e2007a5743dc8378e0ca0253e805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:10.932797 2766 raft_consensus.cc:482] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:10.932996 2766 raft_consensus.cc:504] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.934969 2766 leader_election.cc:288] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d4e2007a5743dc8378e0ca0253e805 (127.16.36.129:40097), 1cc708aef17a498ca22d665643c8f601 (127.16.36.130:39719)
I20240501 14:14:10.950462 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507ea0f416f3403780bab36c07797f77" candidate_uuid: "f381951284524cc5a5b4f5a383b31bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d4e2007a5743dc8378e0ca0253e805" is_pre_election: true
I20240501 14:14:10.950958 2154 raft_consensus.cc:2454] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f381951284524cc5a5b4f5a383b31bb7 in term 0.
I20240501 14:14:10.951539 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507ea0f416f3403780bab36c07797f77" candidate_uuid: "f381951284524cc5a5b4f5a383b31bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc708aef17a498ca22d665643c8f601" is_pre_election: true
I20240501 14:14:10.951624 2352 leader_election.cc:302] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d4e2007a5743dc8378e0ca0253e805, f381951284524cc5a5b4f5a383b31bb7; no voters:
I20240501 14:14:10.951797 2286 raft_consensus.cc:2454] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f381951284524cc5a5b4f5a383b31bb7 in term 0.
I20240501 14:14:10.952092 2766 raft_consensus.cc:2781] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:14:10.952245 2766 raft_consensus.cc:482] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:14:10.952361 2766 raft_consensus.cc:3037] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:14:10.954561 2766 raft_consensus.cc:504] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.955377 2766 leader_election.cc:288] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [CANDIDATE]: Term 1 election: Requested vote from peers 17d4e2007a5743dc8378e0ca0253e805 (127.16.36.129:40097), 1cc708aef17a498ca22d665643c8f601 (127.16.36.130:39719)
I20240501 14:14:10.955879 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507ea0f416f3403780bab36c07797f77" candidate_uuid: "f381951284524cc5a5b4f5a383b31bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d4e2007a5743dc8378e0ca0253e805"
I20240501 14:14:10.956079 2154 raft_consensus.cc:3037] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:14:10.956074 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507ea0f416f3403780bab36c07797f77" candidate_uuid: "f381951284524cc5a5b4f5a383b31bb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc708aef17a498ca22d665643c8f601"
I20240501 14:14:10.956282 2286 raft_consensus.cc:3037] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:14:10.958238 2286 raft_consensus.cc:2454] T 507ea0f416f3403780bab36c07797f77 P 1cc708aef17a498ca22d665643c8f601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f381951284524cc5a5b4f5a383b31bb7 in term 1.
I20240501 14:14:10.958853 2353 leader_election.cc:302] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc708aef17a498ca22d665643c8f601, f381951284524cc5a5b4f5a383b31bb7; no voters:
I20240501 14:14:10.959123 2154 raft_consensus.cc:2454] T 507ea0f416f3403780bab36c07797f77 P 17d4e2007a5743dc8378e0ca0253e805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f381951284524cc5a5b4f5a383b31bb7 in term 1.
I20240501 14:14:10.959198 2766 raft_consensus.cc:2781] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:14:10.959803 2766 raft_consensus.cc:686] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [term 1 LEADER]: Becoming Leader. State: Replica: f381951284524cc5a5b4f5a383b31bb7, State: Running, Role: LEADER
I20240501 14:14:10.960633 2766 consensus_queue.cc:237] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:10.966358 2013 catalog_manager.cc:5484] T 507ea0f416f3403780bab36c07797f77 P f381951284524cc5a5b4f5a383b31bb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f381951284524cc5a5b4f5a383b31bb7 (127.16.36.131). New cstate: current_term: 1 leader_uuid: "f381951284524cc5a5b4f5a383b31bb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } health_report { overall_health: HEALTHY } } }
W20240501 14:14:11.034022 2332 tablet.cc:2369] T 37be6bb30d58488da944702da0c12784 P 1cc708aef17a498ca22d665643c8f601: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:14:11.034057 2463 tablet.cc:2369] T 37be6bb30d58488da944702da0c12784 P f381951284524cc5a5b4f5a383b31bb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:14:11.056200 2200 tablet.cc:2369] T 37be6bb30d58488da944702da0c12784 P 17d4e2007a5743dc8378e0ca0253e805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:14:11.118119 2763 raft_consensus.cc:482] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:11.118424 2763 raft_consensus.cc:504] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:11.119343 2763 leader_election.cc:288] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc708aef17a498ca22d665643c8f601 (127.16.36.130:39719), f381951284524cc5a5b4f5a383b31bb7 (127.16.36.131:46231)
I20240501 14:14:11.119972 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0cd858c23f481087a4e4b12eb42fe8" candidate_uuid: "17d4e2007a5743dc8378e0ca0253e805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f381951284524cc5a5b4f5a383b31bb7" is_pre_election: true
I20240501 14:14:11.119997 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0cd858c23f481087a4e4b12eb42fe8" candidate_uuid: "17d4e2007a5743dc8378e0ca0253e805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc708aef17a498ca22d665643c8f601" is_pre_election: true
I20240501 14:14:11.120235 2417 raft_consensus.cc:2454] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d4e2007a5743dc8378e0ca0253e805 in term 0.
I20240501 14:14:11.120266 2286 raft_consensus.cc:2454] T fe0cd858c23f481087a4e4b12eb42fe8 P 1cc708aef17a498ca22d665643c8f601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d4e2007a5743dc8378e0ca0253e805 in term 0.
I20240501 14:14:11.120734 2090 leader_election.cc:302] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d4e2007a5743dc8378e0ca0253e805, 1cc708aef17a498ca22d665643c8f601; no voters:
I20240501 14:14:11.121084 2763 raft_consensus.cc:2781] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:14:11.121186 2763 raft_consensus.cc:482] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:14:11.121300 2763 raft_consensus.cc:3037] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:14:11.123286 2763 raft_consensus.cc:504] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc708aef17a498ca22d665643c8f601" member_type: VOTER last_known_addr { host: "127.16.36.130" port: 39719 } } peers { permanent_uuid: "17d4e2007a5743dc8378e0ca0253e805" member_type: VOTER last_known_addr { host: "127.16.36.129" port: 40097 } } peers { permanent_uuid: "f381951284524cc5a5b4f5a383b31bb7" member_type: VOTER last_known_addr { host: "127.16.36.131" port: 46231 } }
I20240501 14:14:11.123883 2763 leader_election.cc:288] T fe0cd858c23f481087a4e4b12eb42fe8 P 17d4e2007a5743dc8378e0ca0253e805 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc708aef17a498ca22d665643c8f601 (127.16.36.130:39719), f381951284524cc5a5b4f5a383b31bb7 (127.16.36.131:46231)
I20240501 14:14:11.124563 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0cd858c23f481087a4e4b12eb42fe8" candidate_uuid: "17d4e2007a5743dc8378e0ca0253e805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f381951284524cc5a5b4f5a383b31bb7"
I20240501 14:14:11.124825 2417 raft_consensus.cc:3037] T fe0cd858c23f481087a4e4b12eb42fe8 P f381951284524cc5a5b4f5a383b31bb7 [term 0 FOLLOWER]: Advancing to term 1